modis_GEO.py fails with 0% valid pixels

bbbarnes · by **bbbarnes** » Mon Sep 30, 2019 2:51 pm America/New_York

Perhaps related to this post, but perhaps not...
https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=10951

For the last ~2 weeks, we've had intermittent failures with modis_GEO.py. The program takes a LONG time to run (2-8 hours, depending on the computer), and eventually returns:
Percent valid data (0.00) is less than threshold (95.00)
ERROR: MODIS geolocation processing failed.

Due to similarities to the post above, we updated (install_ocssw.py -b v7.5 --aqua) today, but this did not solve the problem. Only certain files have this issue, and it does not seem to matter if we start processing from L0 or L1A. I've tried on 3 different machines, with no success.

Specifics for one instance (run today after updating ocssw) are below. Note all atteph files were also downloaded today...

> wget https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2005159175000.L1A_LAC.bz2

--2019-09-30 12:15:47-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2005159175000.L1A_LAC.bz2
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 223159889 (213M) [application/octet-stream]
Saving to: â€˜A2005159175000.L1A_LAC.bz2â€™

100%[===============================================================>] 223,159,889 20.1MB/s in 9.1s

2019-09-30 12:15:57 (23.3 MB/s) - â€˜A2005159175000.L1A_LAC.bz2â€™ saved [223159889/223159889]

> bunzip2 A2005159175000.L1A_LAC.bz2

> modis_GEO.py --log -v A2005159175000.L1A_LAC --ancdb=./ancillary.db

Determining required attitude and ephemeris files...
()
Input file: A2005159175000.L1A_LAC
Sensor : modisa
Start time: 2005159175000
End time : 2005159175459
()
Found: /optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2005/159/PM1EPHND.P2005159.1200.001
Found: /optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2005/159/PM1ATTNR.P2005159.1600.003
Found: /optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2005/159/PM1ATTNR.P2005159.1600.002

Creating MODIS geolocation file...
MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 203 Mon Sep 30 12:17:12 2019
scan: 10 out of 203 Mon Sep 30 12:23:05 2019
scan: 20 out of 203 Mon Sep 30 12:29:16 2019
scan: 30 out of 203 Mon Sep 30 12:36:28 2019
scan: 40 out of 203 Mon Sep 30 12:43:19 2019
scan: 50 out of 203 Mon Sep 30 12:50:32 2019
scan: 60 out of 203 Mon Sep 30 12:57:43 2019
scan: 70 out of 203 Mon Sep 30 13:05:00 2019
scan: 80 out of 203 Mon Sep 30 13:12:35 2019
scan: 90 out of 203 Mon Sep 30 13:19:37 2019
scan: 100 out of 203 Mon Sep 30 13:26:55 2019
scan: 110 out of 203 Mon Sep 30 13:34:33 2019
scan: 120 out of 203 Mon Sep 30 13:41:33 2019
scan: 130 out of 203 Mon Sep 30 13:48:41 2019
scan: 140 out of 203 Mon Sep 30 13:56:03 2019
scan: 150 out of 203 Mon Sep 30 14:03:29 2019
scan: 160 out of 203 Mon Sep 30 14:10:46 2019
scan: 170 out of 203 Mon Sep 30 14:18:25 2019
scan: 180 out of 203 Mon Sep 30 14:25:33 2019
scan: 190 out of 203 Mon Sep 30 14:32:18 2019
scan: 200 out of 203 Mon Sep 30 14:39:04 2019
geogen_modis returned with exit status: 0
Percent valid data (0.00) is less than threshold (95.00)
('Validation test returned with error code: ', SystemExit(1,))
ERROR: MODIS geolocation processing failed.

LogUser and LogReport are pretty unremarkable to me (below). LogStatus (attached) is quite long:

> cat LogStatus.A2005159175000.GEO | wc -l

8160

> cat LogUser.A2005159175000.GEO

****************************************
BEGIN_PGE: Mon Sep 30 12:17:12 2019
MSG_TAG: 11
FILE: /tmp/brian_test/twp/new/LogUser.A2005159175000.GEO
LOGGING: status message logging enabled
TRACE_LEVEL: tracing disabled
PID_LOGGING: disabled
DISABLED_LEVELS: none
DISABLED_SEEDS: none
DISABLED_CODES: PGSCSC_W_PREDICTED_UT1 PGSDEM_M_FILLVALUE_INCLUDED
THREAD-SAFE MODE: disabled
TOOLKIT_VERSION: SCF TK5.2.19
****************************************

/home/seadas/ocssw/src/geogen_modis/GEO_location_main.c, main():MODIS_U_GEO_BEGIN:288778279
Mon Sep 30 12:17:12 2019
Seed file: 6.2 Running MOD_PR03 MODIS Geolocation main() 6.2 compiled on Aug 16 2019 at 12:46:24 using makefile 1.1

/home/seadas/ocssw/src/geogen_modis/GEO_location_main.c, main():MODIS_U_GEO_GRANULE_ID:288778281
Mon Sep 30 14:41:23 2019
GRANULE_ID:/tmp/brian_test/twp/new/A2005159175000.L1A_LAC

/home/seadas/ocssw/src/geogen_modis/GEO_location_main.c, main():MODIS_U_GEO_END:288778280
Mon Sep 30 14:41:23 2019
MOD_PR03 controlled exit, with exit code = 0

> cat LogReport.A2005159175000.GEO

****************************************
BEGIN_PGE: Mon Sep 30 12:17:12 2019
MSG_TAG: 11
FILE: /tmp/brian_test/twp/new/LogReport.A2005159175000.GEO
LOGGING: status message logging enabled
TRACE_LEVEL: tracing disabled
PID_LOGGING: disabled
DISABLED_LEVELS: none
DISABLED_SEEDS: none
DISABLED_CODES: PGSCSC_W_PREDICTED_UT1 PGSDEM_M_FILLVALUE_INCLUDED
THREAD-SAFE MODE: disabled
TOOLKIT_VERSION: SCF TK5.2.19
****************************************

Mon Sep 30 14:41:23 2019

Input file name: /tmp/brian_test/twp/new/A2005159175000.L1A_LAC
Output file name: /tmp/brian_test/twp/new/A2005159175000.GEO
Local Granule ID: MYD01.A2005159.1750.005.2010069163447.hdf
scans: 203
pixels: 0
missing: 0
out of bounds: 0
Cumulated gflags: 0, 0, 0, 0, 0, 0, 0, 0
Bounding coordinates
east: 3.141593
west: -3.141593
north: -1.570796
south: 1.570796
==========================================================

Advice?
attachment 1

by **OB.DAAC - SeanBailey** » Mon Sep 30, 2019 4:05 pm America/New_York

Brian,
The problem is that two attitude files are retreived, when only one should be:
PM1ATTNR.P2005159.1600.002
PM1ATTNR.P2005159.1600.003

The earilier version (002) is set to "QCFAIL" in our production DB. That *should* prevent it from being selected (and does in our production), but something is amiss in the selection rules for the modis_atteph.py script distributed with SeaDAS ( or rather with the DB calls used by the backend of that script). May take a bit of navel gazing to resolve this one. I'll let you know when it's fixed.

In the meantime, if you find such cases, you can eliminate the '002' files and it will work, e.g.

modis_GEO.py -v --att1=$ANCDIR/2005/159/PM1ATTNR.P2005159.1600.003 --eph1=$ANCDIR/2005/159/PM1EPHND.P2005159.1200.001 A2005159175000.L1A_LAC

Sean

gnwiii · by **gnwiii** » Mon Sep 30, 2019 4:35 pm America/New_York

This looks like a different problem. I can reproduce the problem using the same L1A file on a system where modis_GEO.py works for the SeaDAS benchmark file (A2006167181000.L1A). In the attached LogStatus file you can see a hint:

PGS_EPH_getAttitRecords():PGS_E_TOOLKIT:3850
 Too many recursive calls of PGS_EPH_getAttitRecords(), Ephemeris/Attitude data file may be corrupted

PGS_EPH_getAttitRecords():PGS_E_TOOLKIT - 2708 additional occurrences

PGS_EPH_EphemAttit():PGS_E_TOOLKIT:3850
 Too many recursive calls of PGS_EPH_getAttitRecords(), Ephemeris/Attitude data file may be corrupted

This is a different error and may explain why the program takes so long.

by **OB.DAAC - SeanBailey** » Mon Sep 30, 2019 7:08 pm America/New_York

The modis_GEO.py script will work if the set of attitude and ephemeris data provided to it are correct. There are a number of such files for which we have updated versions. For reasons I have yet had time to investigate, both old and new files are returned and used by the script, which confuses the geogen_modis code. Only the updated version should be returned. Once that problem is resolved, modis_GEO.py will be happy again...for all files :grin:

Sean

bbbarnes · by **bbbarnes** » Mon Sep 30, 2019 9:50 pm America/New_York

Thanks Sean and George. Until you're able to resolve this on your end, we can modify our processing stream to check for 2 att (or eph) files returning from modis_atteph.py. In such instances, we'll use the one with the higher version number in the modis_GEO.py call.

by **OB.DAAC - SeanBailey** » Tue Oct 01, 2019 8:50 am America/New_York

Fixed. After sufficient gazing, the solution presented itself - and it was a back-end fix, so nothing for you to do except run as normal.

Sean

gnwiii · by **gnwiii** » Tue Oct 01, 2019 3:18 pm America/New_York

Thanks. I can confirm that "run as normal" works for the problem file:

$ modis_GEO.py --log -v A2005159175000.L1A_LAC --ancdb=./ancillary.db
[...]
Determining required attitude and ephemeris files...
()
Input file: A2005159175000.L1A_LAC
Sensor    : modisa
Start time: 2005159175000
End time  : 2005159175459
()
  Found: /home/sdsuser/ocssw/var/anc/2005/159/PM1EPHND.P2005159.1200.001
  Found: /home/sdsuser/ocssw/var/anc/2005/159/PM1ATTNR.P2005159.1600.003

Creating MODIS geolocation file...
MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 203 Tue Oct  1 16:12:15 2019
[...]
Percentage of pixels with missing geolocation: 0.00
Validation test passed for geolocation file A2005159175000.GEO
geogen_modis created A2005159175000.GEO successfully!
MODIS geolocation processing complete.

bbbarnes · by **bbbarnes** » Wed Oct 02, 2019 11:55 am America/New_York

Thanks, Sean. Can confirm this works on our end now, too.