processing l0 to l1a fails

Please enter here to ask a question about any NASA Science related topics!
oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

processing l0 to l1a fails

by oo_processing » Thu Dec 10, 2020 1:11 pm America/New_York

I am having issues processing L0 data (both by hand and in automated jobs). Please advise.

[bmurch@optics0 test]$  modis_L1A.py --verbose MOD00.P2020345.0650_1.PDS -o MOD00.P2020345.0650.BIGBEND_FULL.L1A_LAC --startnudge=0 --stopnudge=10
Determining pass start and stop time...                                                                                                          

Using specified output L1A filename: MOD00.P2020345.0650.BIGBEND_FULL.L1A_LAC
('Input Level 0:', 'MOD00.P2020345.0650_1.PDS')                             
('Output Level 1A:', 'MOD00.P2020345.0650.BIGBEND_FULL.L1A_LAC')            
('Satellite:', 'aqua')                                                      
('Start Time:', '2020-12-10T06:50:00.000000Z')                              
('Stop Time:', '2020-12-10T06:54:50.000000Z')                               
('Granule Duration:', 290.0, 'seconds')                                     

Processing MODIS L0 file to L1A...
MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26
scan: 0 out of 201 Thu Dec 10 13:01:43 2020       
...     
scan: 190 out of 201 Thu Dec 10 13:01:54 2020     
('l1agen_modis exit status:', '0')                
MODIS L1A processing complete.  
                 
[bmurch@optics0 test]$  modis_atteph.py MOD00.P2020345.0650.BIGBEND_FULL.L1A_LAC
Aqua                                                                           
Aqua                                                                           
att1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR.P2020345.0600.003
eph1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0645.061
eph2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0650.061
eph3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0655.061
[bmurch@optics0 test]$ modis_GEO.py --verbose MOD00.P2020345.0650.BIGBEND_FULL.L1A_LAC -o MOD00.P2020345.0650.BIGBEND_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
Determining required attitude and ephemeris files...                                                                                                                      
()
Input file: MOD00.P2020345.0650.BIGBEND_FULL.L1A_LAC
Sensor    : modisa
Start time: 2020345065000
End time  : 2020345065450
()
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0655.061
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0650.061
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0645.061
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR.P2020345.0600.003


Creating MODIS geolocation file...
MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 196 Thu Dec 10 13:04:31 2020
...
scan: 190 out of 196 Thu Dec 10 13:06:30 2020
geogen_modis returned with exit status: 1
Percent valid data (0.00) is less than threshold (95.00)
('Validation test returned with error code: ', SystemExit(1,))
ERROR: MODIS geolocation processing failed.

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1219
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

processing l0 to l1a fails

by OB.DAAC - SeanBailey » Thu Dec 10, 2020 1:32 pm America/New_York

Brock,

When geolocation fails, it leaves around some log files that may contain some helpful information (LogStatus, LogUser and LogReport).  Check them.  If you can't decipher them, attach (don't copy/paste inline) them to a reply post.  I know this may not be of much comfort to you but... works for me.

Sean

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

processing l0 to l1a fails

by oo_processing » Thu Dec 10, 2020 1:39 pm America/New_York

Sean,

There were no such files (but I learned my lesson on posting whole logs long ago -- sometimes I slip and forget and post a snippet), however, it appears that the problem is this:

/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1ATTN0.A2020345.0000.002:     HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1ATTN0.A2020345.0200.002:     HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1ATTN0.A2020345.0600.002:     HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1EPHN0.A2020345.0000.002:     HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1EPHN0.A2020345.0200.002:     HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1EPHN0.A2020345.0600.002:     HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0105.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0110.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0250.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0255.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0430.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0435.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0440.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0445.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0610.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0615.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0640.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0645.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0650.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0655.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0955.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.1000.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0105.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0110.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0250.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0255.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0430.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0640.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0645.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0650.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0655.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0955.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.1000.061: HTML document text

fredoceansips
Posts: 7
Joined: Fri Nov 09, 2007 1:39 pm America/New_York

processing l0 to l1a fails

by fredoceansips » Thu Dec 10, 2020 1:56 pm America/New_York

The most likely cause is that you were not logged into Earthdata when you downloaded the attitude and ephemeris files using wget.  When you do this the downloaded file is an HTML rather than the requested file, but it still has the same name.  Yes, this is not the most informative response.

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1219
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

processing l0 to l1a fails

by OB.DAAC - SeanBailey » Thu Dec 10, 2020 1:57 pm America/New_York

Yep, that'd be a problem.  Probably want to delete those and reacquire :wink:

Sean
Edit: BTW, if the log files don't get saved, you can rerun with the --log option to prevent them from being deleted.

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

processing l0 to l1a fails

by oo_processing » Thu Dec 10, 2020 4:05 pm America/New_York

I'm having a similar issue as the one caused by the file that needed to be replaced on Nov 30 ( There are 3.14 errors in the log files in the attached files):

NOTE that this failure occurs on two separate installations.

Input file name: /tmp/test/MOD00.P2020345.1000.GLAKES_FULL.L1A_LAC
Output file name: /tmp/test/MOD00.P2020345.1000.GLAKES_FULL.GEO  
Local Granule ID: MYD01.A2020345.1000.061.2020345205056.hdf      
scans: 196                                                       
pixels: 0                                                        
        missing: 0                                               
        out of bounds: 0                                         
Cumulated gflags: 0, 0, 0, 0, 0, 0, 0, 0                         
Bounding coordinates                                             
        east: 3.141593                                           
        west: -3.141593                                          
        north: -1.570796                                         
        south: 1.570796

searobin 162 :/tmp/test> modis_L1A.py MOD00.P2020345.1000_1.PDS  -o ./MOD00.P2020345.1000.GLAKES_FULL.L1A_LAC --startnudge=0 --stopnudge=10                       
MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26                                                                                                               
scan: 0 out of 201 Thu Dec 10 15:50:55 2020                                                                                                                       
...                                                                                                                  
scan: 190 out of 201 Thu Dec 10 15:50:56 2020                                                                                                                     
searobin 163 :/tmp/test> modis_GEO.py --verbose ./MOD00.P2020345.1000.GLAKES_FULL.L1A_LAC -o ./MOD00.P2020345.1000.GLAKES_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
Determining required attitude and ephemeris files...                                                                                                                        
Searching database: ./ancillary_data.db                                                                                                                                     
()                                                                                                                                                                          
Input file: ./MOD00.P2020345.1000.GLAKES_FULL.L1A_LAC                                                                                                                       
Sensor    : modisa                                                                                                                                                          
Start time: 2020345100000                                                                                                                                                   
End time  : 2020345100450                                                                                                                                                   
()                                                                                                                                                                          
  Found: /optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.1005.061                                                                        
  Found: /optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.1000.061                                                                        
  Found: /optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0955.061                                                                        
  Found: /optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR.P2020345.0800.003                                                                            

Creating MODIS geolocation file...
MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 196 Thu Dec 10 15:51:41 2020       
...  
scan: 190 out of 196 Thu Dec 10 15:51:41 2020     
geogen_modis returned with exit status: 0         
Percent valid data (0.00) is less than threshold (95.00)
('Validation test returned with error code: ', SystemExit(1,))
ERROR: MODIS geolocation processing failed.
attachment 1

attachment 2

attachment 3

attachment 4

woodbri
Posts: 52
Joined: Thu Jun 04, 2015 10:50 am America/New_York

processing l0 to l1a fails

by woodbri » Thu Dec 10, 2020 7:03 pm America/New_York

I'm also getting failures today.

## /u/oceancolor-bin/parallel-modis-seadas  MOD00.P2020345.1930_1.PDS.bz2

modis_L1A.py MOD00.P2020345.1930_1.PDS -o MOD00.P2020345.1930_1.PDS.L1A_LAC
MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26
scan: 0 out of 207 Thu Dec 10 18:44:09 2020
...
scan: 200 out of 207 Thu Dec 10 18:44:10 2020

getanc.py --refreshDB MOD00.P2020345.1930_1.PDS.L1A_LAC
Aqua
icefile=/u/oceandata/var/anc/2020/344/N202034400_SEAICE_NSIDC_24h.hdf
met1=/u/oceandata/var/anc/2020/345/N202034512_MET_NCEP_6h.hdf
met2=/u/oceandata/var/anc/2020/345/N202034512_MET_NCEP_6h.hdf
met3=/u/oceandata/var/anc/2020/345/N202034512_MET_NCEP_6h.hdf
sstfile=/u/oceandata/var/anc/2020/344/N2020344_SST_OIV2AV_24h.nc

()
*** WARNING: The following ancillary data types were missing or are not optimal:  MET OZONE
*** Beware that certain MET and OZONE files just chosen by this program are not optimal.
*** For near real-time processing the remaining files may become available soon.

Failed: 'getanc.py --refreshDB MOD00.P2020345.1930_1.PDS.L1A_LAC', error 768

modis_atteph.py --refreshDB -v MOD00.P2020345.1930_1.PDS.L1A_LAC
Aqua
Searching database: /u/oceandata/var/ancillary_data.db
Determining pass start and end times...
Aqua
()
Input file: MOD00.P2020345.1930_1.PDS.L1A_LAC
Sensor    : aqua
Start time: 2020345193000
End time  : 2020345193500
()
OBPG session started
  Found: /u/oceandata/var/anc/2020/345/PM1ATTNR_NRT.A2020345.1935.061
  Found: /u/oceandata/var/anc/2020/345/PM1ATTNR_NRT.A2020345.1930.061
  Found: /u/oceandata/var/anc/2020/345/PM1ATTNR_NRT.A2020345.1925.061
  Found: /u/oceandata/var/anc/2020/345/PM1EPHND_NRT.A2020345.1935.061
  Found: /u/oceandata/var/anc/2020/345/PM1EPHND_NRT.A2020345.1930.061
  Found: /u/oceandata/var/anc/2020/345/PM1EPHND_NRT.A2020345.1925.061
All required attitude and ephemeris files successfully determined and downloaded.
att1=/u/oceandata/var/anc/2020/345/PM1ATTNR_NRT.A2020345.1925.061
att2=/u/oceandata/var/anc/2020/345/PM1ATTNR_NRT.A2020345.1930.061
att3=/u/oceandata/var/anc/2020/345/PM1ATTNR_NRT.A2020345.1935.061
eph1=/u/oceandata/var/anc/2020/345/PM1EPHND_NRT.A2020345.1925.061
eph2=/u/oceandata/var/anc/2020/345/PM1EPHND_NRT.A2020345.1930.061
eph3=/u/oceandata/var/anc/2020/345/PM1EPHND_NRT.A2020345.1935.061

()
- All optimal ancillary data files were determined and downloaded. -

modis_GEO.py MOD00.P2020345.1930_1.PDS.L1A_LAC -o MOD00.P2020345.1930_1.PDS.GEO
MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 203 Thu Dec 10 18:44:14 2020
...
scan: 200 out of 203 Thu Dec 10 18:44:14 2020
Percent valid data (0.00) is less than threshold (95.00)
ERROR: MODIS geolocation processing failed.

Failed: 'modis_GEO.py MOD00.P2020345.1930_1.PDS.L1A_LAC -o MOD00.P2020345.1930_1.PDS.GEO', error 256
ERROR: Failed to run '/u/oceancolor-bin/parallel-modis-seadas  MOD00.P2020345.1930_1.PDS.bz2' with exit 256
parallel-modis-aqua.pl: Problems processing MOD00.P2020345.1930_1.PDS.bz2 in /maps/nasa/tmp-17427

And similar failures for all files today.

woodbri
Posts: 52
Joined: Thu Jun 04, 2015 10:50 am America/New_York

processing l0 to l1a fails

by woodbri » Thu Dec 10, 2020 7:32 pm America/New_York

And it looks like it has been failing for few days back. Did something change on the data? Did i miss an announcement and do I need to update seadas software? I'm on branch origin/v7.5

gnwiii
Posts: 641
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

processing l0 to l1a fails

by gnwiii » Thu Dec 10, 2020 8:20 pm America/New_York

Works for me.  Did you see https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=12848?  

% ls -l $OCDATAROOT/modis/static/de200*
-rw-r--r--. 1 seadas seadas 17701056 Sep 24  2019 /home/seadas/ocssw/share/modis/static/de200.dat
-rw-r--r--. 1 seadas seadas 15961464 Dec 10 21:00 /home/seadas/ocssw/share/modis/static/de200.eos
-rw-r--r--. 1 seadas seadas  4767192 Sep 24  2019 /home/seadas/ocssw/share/modis/static/de200.eos.old

woodbri
Posts: 52
Joined: Thu Jun 04, 2015 10:50 am America/New_York

processing l0 to l1a fails

by woodbri » Thu Dec 10, 2020 8:47 pm America/New_York

Thank you! I think that fixes my problem. I saw that notice, but don't know why I didn't respond to it immediately, then total forgot! Duh ... hand to forehead!

Post Reply