Pass processed NRT OK, reprocess 5 days later fails?

Please enter here to ask a question about any NASA Science related topics!
Post Reply
oo_processing
Posts: 206
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

Pass processed NRT OK, reprocess 5 days later fails?

by oo_processing » Tue Mar 10, 2020 1:54 pm America/New_York

Sean,
Sorry for the long post. But... I ran this combined granule NRT on 2020-03-05 13:53:07 , and it failed as happens when finding a new way to do things. I
I ran it again at 2020-03-05 15:02:38 (still NRT) and it was was successful. However, I reprocess things days later, and that failed at the place where the NRT processing worked fine. See bolded below:

Here is the good processing from 3/5
STATUS: $ocssw_command_1 = modis_L1A.py /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.PDS -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC --startnudge=0 --stopnudge=10
STATUS: $ocssw_return_1 = MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26
scan: 0 out of 404 Thu Mar  5 15:03:06 2020
...
scan: 390 out of 404 Thu Mar  5 15:03:09 2020
STATUS: $l1a_file_w_vdir_path /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC was found
$_: att1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1550.061
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1550.061 was not compressed
STATUS: $compression_test_returns existing file: 1
$_:  att2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1555.061
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1555.061 was not compressed
STATUS: $compression_test_returns existing file: 1
$_:  att3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1600.061
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1600.061 was not compressed
STATUS: $compression_test_returns existing file: 1
$_:  att4=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1605.061
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1605.061 was not compressed
STATUS: $compression_test_returns existing file: 1
$_:  eph1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1550.061
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1550.061 was not compressed
STATUS: $compression_test_returns existing file: 1
$_:  eph2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1555.061
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1555.061 was not compressed
STATUS: $compression_test_returns existing file: 1
$_:  eph3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1600.061
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1600.061 was not compressed
STATUS: $compression_test_returns existing file: 1
$_:  eph4=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1605.061
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1605.061 was not compressed
STATUS: $compression_test_returns existing file: 1
$_: icefile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/063/N202006300_SEAICE_NSIDC_24h.hdf
$_:  met1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEP_6h.hdf
$_:  met2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEP_6h.hdf
$_:  met3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEP_6h.hdf
$_:  sstfile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/064/N2020064_SST_OIV2AV_24h.nc

STATUS: $ocssw_command_2 = modis_GEO.py --parfile=./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
STATUS: $ocssw_return_2 = MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 400 Thu Mar  5 15:03:16 2020
...
scan: 390 out of 400 Thu Mar  5 15:03:42 2020
./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed
{'geogen': {}}
STATUS: The command: modis_GEO.py --parfile=./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db RETURNED NO ERROR message

This next bit is the days later reprocessing log snippet I just ran 2020-03-10 12:40:06:
STATUS: $ocssw_command_1 = modis_L1A.py /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.PDS -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC --startnudge=0 --stopnudge=10
[56208 refs]
STATUS: $ocssw_return_1 = MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26
scan: 0 out of 404 Tue Mar 10 12:40:45 2020
...
scan: 390 out of 404 Tue Mar 10 12:40:47 2020
$_: att1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003 was not compressed
STATUS: $compression_test_returns existing file: 1
$_:  att2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003 was not compressed
STATUS: $compression_test_returns existing file: 1
$_:  eph1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003 was not compressed
STATUS: $compression_test_returns existing file: 1
[66172 refs]
$_: icefile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_SEAICE_NSIDC_24h.hdf
$_:  met1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEPR2_6h.hdf
$_:  met2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf
$_:  met3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf
$_:  ozone1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_O3_AURAOMI_24h.hdf
$_:  ozone2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf
$_:  ozone3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf
$_:  sstfile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N2020065_SST_OIV2AV_24h.nc

STATUS: $ocssw_command_2 = modis_GEO.py --parfile=./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
[66403 refs]
STATUS: $ocssw_return_2 = MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 400 Tue Mar 10 12:40:49 2020
...
scan: 390 out of 400 Tue Mar 10 12:41:00 2020
./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed
{'geogen': {}}
Percent valid data (34.00) is less than threshold (95.00)
ERROR: MODIS geolocation processing failed.
ERROR: The command: modis_GEO.py --parfile=./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
returned an error code 'ERROR:'
and likely 'ERROR: MODIS geolocation processing failed.'


I have looked at the anc files and they appear correct:
[bmurch@optics0 cron]$ file /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_SEAICE_NSIDC_24h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEPR2_6h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_O3_AURAOMI_24h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N2020065_SST_OIV2AV_24h.nc
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003:      data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003:      data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003:      data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_SEAICE_NSIDC_24h.hdf: Hierarchical Data Format (version 4) data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEPR2_6h.hdf:    Hierarchical Data Format (version 4) data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf:    Hierarchical Data Format (version 4) data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf:    Hierarchical Data Format (version 4) data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_O3_AURAOMI_24h.hdf:   Hierarchical Data Format (version 4) data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf:   Hierarchical Data Format (version 4) data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf:   Hierarchical Data Format (version 4) data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N2020065_SST_OIV2AV_24h.nc:      Hierarchical Data Format (version 5) data

ls -l returns
-rw-rw-r-- 1 bmurch cms_optics  201285 Mar  7 06:56 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_O3_AURAOMI_24h.hdf
-rw-rw-r-- 1 bmurch cms_optics  245446 Mar  6 18:12 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_SEAICE_NSIDC_24h.hdf
-rw-rw-r-- 1 bmurch cms_optics 1650473 Mar 10 00:11 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEPR2_6h.hdf
-rw-rw-r-- 1 bmurch cms_optics 1650461 Mar 10 00:11 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf
-rw-rw-r-- 1 bmurch cms_optics 1650461 Mar 10 00:11 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf
-rw-rw-r-- 1 bmurch cms_optics 1656193 Mar  6 07:51 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N2020065_SST_OIV2AV_24h.nc
-rw-rw-r-- 1 bmurch cms_optics  462848 Mar 10 00:14 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003
-rw-rw-r-- 1 bmurch cms_optics  462848 Mar 10 00:14 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003
-rw-rw-r-- 1 bmurch cms_optics 5531104 Mar 10 12:54 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003
-rw-rw-r-- 1 bmurch cms_optics  201285 Mar  8 07:55 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf
-rw-rw-r-- 1 bmurch cms_optics  201285 Mar  8 07:55 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1166
Joined: Wed Sep 18, 2019 6:15 pm America/New_York

Pass processed NRT OK, reprocess 5 days later fails?

by OB.DAAC - SeanBailey » Wed Mar 11, 2020 8:38 am America/New_York

Brock,
The NRT attitude and ephemeris files are predicted data, predictions don't know about reality.
In this case the reality is that MODIS-Aqua executed a lunar roll maneuver on day 065, information of which the defninite attitude and ephemeris have knowledge:
https://mcst.gsfc.nasa.gov/modis-events-categories/routine-calibrations?event_date%5Bvalue%5D%5Byear%5D=&modis_mission=60

Interestingly, though, the maneuver was later in the day...but something was up, as the data indicate the spacecraft wasn't exactly stable.  These files are nighttime data, the image below is of the 11um channel (band 31).  Even if geolcation was successful, I doubt the data would be useful to you...

Sean

Post Reply