Some modis_GEO.py 7.5.3 fail but not in modis_GEO T2022.2 anc.db issues

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Some modis_GEO.py 7.5.3 fail but not in modis_GEO T2022.2 anc.db issues

by oo_processing » Wed Mar 02, 2022 3:22 pm America/New_York

I use 7.5.3 in production as there are issues using newer code that need to me resolved.
This is not longer true again:
viewtopic.php?p=10413&hilit=getanc+retu ... 327#p10411

I have had some files, __not_all__, fail in the geo creation with 7.5.3 and not in T2022.2, using the same files in the same directory.

From 7.5.3:
[bmurch@optics0 S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing]$ pwd
/work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing
[bmurch@optics0 S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing]$ which modis_GEO.py
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/scripts/modis_GEO.py
[bmurch@optics0 S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing]$ modis_GEO.py MOD00.P2022061.0745.MRIVER_FULL.L1A_LAC -o MOD00.P2022061.0745.MRIVER_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 197 Wed Mar 2 15:11:04 2022
...
scan: 190 out of 197 Wed Mar 2 15:11:06 2022
Percent valid data (0.00) is less than threshold (95.00)
ERROR: MODIS geolocation processing failed.
[bmurch@optics0 S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing]$ strings ./ancillary_data.db
SQLite format 3
Wtablesatancinfosatancinfo
CREATE TABLE satancinfo
(satid INTEGER ,
ancid INTEGER ,
optimal INTEGER,
FOREIGN KEY(satID) REFERENCES satfiles(satid),
FOREIGN KEY(ancID) REFERENCES ancfiles(ancid))
tableancfilesancfiles
CREATE TABLE ancfiles
(ancid INTEGER PRIMARY KEY,
filename TEXT ,
path TEXT ,
type TEXT)
%tablesatfilessatfiles
CREATE TABLE satfiles
(satid INTEGER PRIMARY KEY,
filename TEXT ,
starttime TEXT,
stoptime TEXT,
status INTEGER,
attephstat INTEGER)
MOD00.P2022061.0745.MRIVER_FULL.L1A_LAC20220610745002022061074950
PM1ATTNR.P2022061.0600.003/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/061att1j
PM1EPHND_NRT.A2022061.0750.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/061eph1


From T2022.2:
[bmurch@optics0 S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing]$ pwd
/work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing
[bmurch@optics0 S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing]$ modis_GEO MOD00.P2022061.0745.MRIVER_FULL.L1A_LAC -o MOD00.P2022061.0745.MRIVER_FULL.GEO --threshold=95 --ancdb=./ancillary_data_T2022.2.db
./ancillary_data_T2022.2.db
MODIS GEO version 6.1.0, built Feb 9 2022 11:41:17
scan: 0 out of 197 Wed Mar 2 15:14:54 2022
...
scan: 190 out of 197 Wed Mar 2 15:15:02 2022
[bmurch@optics0 S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing]$ strings ./ancillary_data_T2022.2.db
SQLite format 3
Wtablesatancinfosatancinfo
CREATE TABLE satancinfo
(satid INTEGER ,
ancid INTEGER ,
optimal INTEGER,
FOREIGN KEY(satID) REFERENCES satfiles(satid),
FOREIGN KEY(ancID) REFERENCES ancfiles(ancid))
tableancfilesancfiles
CREATE TABLE ancfiles
(ancid INTEGER PRIMARY KEY,
filename TEXT ,
path TEXT ,
type TEXT)
%tablesatfilessatfiles
CREATE TABLE satfiles
(satid INTEGER PRIMARY KEY,
filename TEXT ,
starttime TEXT,
stoptime TEXT,
status INTEGER,
attephstat INTEGER)
MOD00.P2022061.0745.MRIVER_FULL.L1A_LAC2022-03-02T07:45:002022-03-02T07:49:50
N2022060_SST_OIV2AV_24h.nc/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/c/sstfilec
N202206000_SEAICE_NSIDC_24h.hdf/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/df/icefile`
N202205900_O3_AURAOMI_24h.hdf/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/df/ozone3`
N202205900_O3_AURAOMI_24h.hdf/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/df/ozone2`
N202205900_O3_AURAOMI_24h.hdf/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/df/ozone1[
N202206112_MET_NCEP_6h.hdf/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/df/met3[
N202206112_MET_NCEP_6h.hdf/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/df/met2[
N202206106_MET_NCEP_6h.hdf/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/df/met1a
PM1EPHND.P2022060.1200.003/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/060eph1a
PM1ATTNR.P2022061.0600.003/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/061att1
Last edited by oo_processing on Wed Mar 02, 2022 4:05 pm America/New_York, edited 1 time in total.

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Re: Some modis_GEO.py 7.5.3 fail but not in modis_GEO T2022.2 anc.db issues

by oo_processing » Wed Mar 02, 2022 3:57 pm America/New_York

It appears that these are the ones having issues right now (there may be more that I don't use):

FILE=/cms_zfs/sat_data/modis/l0/2022/061/MOD00.P2022061.0740_1.PDS.bz2
FILE=/cms_zfs/sat_data/modis/l0/2022/061/MOD00.P2022061.0745_1.PDS.bz2
FILE=/cms_zfs/sat_data/modis/l0/2022/061/MOD00.P2022061.0750_1.PDS.bz2
FILE=/cms_zfs/sat_data/modis/l0/2022/061/MOD00.P2022061.0850_1.PDS.bz2
FILE=/cms_zfs/sat_data/modis/l0/2022/061/MOD00.P2022061.0855_1.PDS.bz2

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Re: Some modis_GEO.py 7.5.3 fail but not in modis_GEO T2022.2 anc.db issues

by oo_processing » Wed Mar 02, 2022 4:57 pm America/New_York

Also note that 7.5.3 returned a correct _NRT, but the T2022.2 version points to a day old 060/PM1EPHND.P2022060.1200.003.
I'm not sure why that would be the case if the _NRT is available as demonstrated in 7.5.3 DB?

[bmurch@optics0 seadas_l1a_geo_extract_rt]$ ll /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/061/PM1EPHND_NRT.A2022061.0750.061
-rw-rw-r-- 1 bmurch cms_optics 20000 Mar 2 08:30 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/061/PM1EPHND_NRT.A2022061.0750.061
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ ll /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/060/PM1EPHND.P2022060.1200.003
-rw-rw-r-- 1 bmurch cms_optics 5531104 Mar 2 14:44 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/060/PM1EPHND.P2022060.1200.003

Also note that this file exists in the test directory
[bmurch@optics0 S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing]$ cat MOD00.P2022061.0745.MRIVER_FULL.L1A_LAC.anc
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/060/N202206000_SEAICE_NSIDC_24h.hdf
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/061/N202206106_MET_NCEP_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/061/N202206112_MET_NCEP_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/061/N202206112_MET_NCEP_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/059/N202205900_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/059/N202205900_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/059/N202205900_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2022/060/N2022060_SST_OIV2AV_24h.nc

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Re: Some modis_GEO.py 7.5.3 fail but not in modis_GEO T2022.2 anc.db issues

by OB.DAAC - SeanBailey » Thu Mar 03, 2022 1:30 pm America/New_York

Brock,

There was (still is until we can correct it) some missing metadata in our database related to some ancillary data. That was confusing the getanc script, having it pull a NRT ephemeris file instead of the definitive one. The newer getanc uses a different (and more robust) mechanism so was unaffected by this issue. You should use the newer one. Really...once we complete our pending reprocessing, the older one will not return the correct ancillary files....

You can use the newer getanc with the older SeaDAS version (but you probably want to figure out the issues you have with the newer and update your full SeaDAS implementation).

BTW, the definitive ephemeris are 24hr files, so for the scene in question, the ephemeris file with a name that suggests the day prior is still the correct file.

Sean

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Re: Some modis_GEO.py 7.5.3 fail but not in modis_GEO T2022.2 anc.db issues

by oo_processing » Thu Mar 03, 2022 6:44 pm America/New_York

The files finally went though. I don't know why?
The content of the db that was sucessful was
[bmurch@optics0 S4P_MODIS_H5_MODISA.MRIVER.2022.061_testing]$ strings ancillary_data-7.5.3.db
SQLite format 3
Wtablesatancinfosatancinfo
CREATE TABLE satancinfo
(satid INTEGER ,
ancid INTEGER ,
optimal INTEGER,
FOREIGN KEY(satID) REFERENCES satfiles(satid),
FOREIGN KEY(ancID) REFERENCES ancfiles(ancid))
tableancfilesancfiles
CREATE TABLE ancfiles
(ancid INTEGER PRIMARY KEY,
filename TEXT ,
path TEXT ,
type TEXT)
%tablesatfilessatfiles
CREATE TABLE satfiles
(satid INTEGER PRIMARY KEY,
filename TEXT ,
starttime TEXT,
stoptime TEXT,
status INTEGER,
attephstat INTEGER)
MOD00.P2022061.0745.MRIVER_FULL.L1A_LAC20220610745002022061074950
PM1ATTNR.P2022061.0600.003/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/061att1j
PM1EPHND_NRT.A2022061.0740.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/061eph1j
PM1EPHND_NRT.A2022061.0745.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/061eph2j
PM1EPHND_NRT.A2022061.0750.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/061eph3

Post Reply