T2021.24 scripts not returning correctly

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
oo_processing
Posts: 240
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

T2021.24 scripts not returning correctly

by oo_processing » Fri Jan 21, 2022 6:03 pm America/New_York

So it appears that a problem (well for us anyways) remain from way, way back.

The old one is that output of getanc and modis_atteph return att1 att2 ... eph1 eph2
BUT the other program takes a1 a2 ... e1 e2

And worse, the 2nd issue is that the att and eph files returned by getanc DO NOT PUT THE PATH INTO THE RETURN.
Yet, modis_atteph returns the att and eph files with the path? (AND SEE BELOW and note the OZONE warning)

New ancDBs were used for testing both 7.5.3 and 8.1 (T2021.24)
[bmurch@mdc-1057-13-5]$ getanc ./MOD00.P2022001.1820.CWFL_FULL.L1A_LAC --ancdb ./ancillary_data.db
./ancillary_data.db

Aqua
*** WARNING: No optimal OZONE files found.
att1=PM1ATTNR.P2022001.1800.003
eph1=PM1EPHND.P2022001.1200.003
icefile=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/001/N2022001_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/001/N202200118_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/002/N202200200_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/002/N202200200_MET_NCEPR2_6h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/001/N2022001_SST_OIV2AV_24h.nc


*** WARNING: The following ancillary data types were missing or are not optimal: OZONE

Whereas this places a path:
[bmurch@mdc-1057-13-5]$ modis_atteph MOD00.P2022001.1820.CWFL_FULL.L1A_LAC --ancdb ./ancillary_data.db Aqua
./ancillary_data.db

Aqua
att1=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/001/PM1ATTNR.P2022001.1800.003
eph1=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/001/PM1EPHND.P2022001.1200.003
icefile=N2022001_SST_OIV2AV_24h.nc
met1=N202200118_MET_NCEPR2_6h.hdf
met2=N202200200_MET_NCEPR2_6h.hdf
met3=N202200200_MET_NCEPR2_6h.hdf
sstfile=N2022001_SST_OIV2AV_24h.nc

But even worse? Maybe? Is that 7.5.3 finds the ozone files without issue:

getanc.py ./MOD00.P2022001.1820.CWFL_FULL.L1A_LAC --ancdb ./ancillary_data_753.db
Aqua
icefile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/001/N202200100_SEAICE_NSIDC_24h.hdf
met1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/001/N202200118_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/002/N202200200_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/002/N202200200_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/001/N202200100_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/002/N202200200_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/002/N202200200_O3_AURAOMI_24h.hdf

sstfile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/001/N2022001_SST_OIV2AV_24h.nc
[bmurch@mdc-1057-13-5] $ strings ./ancillary_data_753.db
SQLite format 3
Wtablesatancinfosatancinfo
CREATE TABLE satancinfo
(satid INTEGER ,
ancid INTEGER ,
optimal INTEGER,
FOREIGN KEY(satID) REFERENCES satfiles(satid),
FOREIGN KEY(ancID) REFERENCES ancfiles(ancid))
tableancfilesancfiles
CREATE TABLE ancfiles
(ancid INTEGER PRIMARY KEY,
filename TEXT ,
path TEXT ,
type TEXT)
%tablesatfilessatfiles
CREATE TABLE satfiles
(satid INTEGER PRIMARY KEY,
filename TEXT ,
starttime TEXT,
stoptime TEXT,
status INTEGER,
attephstat INTEGER)
MOD00.P2022001.1820.CWFL_FULL.L1A_LAC20220011820002022001182450
N202200100_SEAICE_NSIDC_24h.hdf/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/001icefilei
N2022001_SST_OIV2AV_24h.nc/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/001sstfilek
N202200200_O3_AURAOMI_24h.hdf/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/002ozone2k
N202200200_O3_AURAOMI_24h.hdf/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/002ozone3k
N202200100_O3_AURAOMI_24h.hdf/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/001ozone1h

N202200118_MET_NCEPR2_6h.hdf/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/001met1h
N202200200_MET_NCEPR2_6h.hdf/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/002met3h
N202200200_MET_NCEPR2_6h.hdf/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2022/002met2

VS

$ strings ancillary_data.db
SQLite format 3
Wtablesatancinfosatancinfo
CREATE TABLE satancinfo
(satid INTEGER ,
ancid INTEGER ,
optimal INTEGER,
FOREIGN KEY(satID) REFERENCES satfiles(satid),
FOREIGN KEY(ancID) REFERENCES ancfiles(ancid))
tableancfilesancfiles
CREATE TABLE ancfiles
(ancid INTEGER PRIMARY KEY,
filename TEXT ,
path TEXT ,
type TEXT)
%tablesatfilessatfiles
CREATE TABLE satfiles
(satid INTEGER PRIMARY KEY,
filename TEXT ,
starttime TEXT,
stoptime TEXT,
status INTEGER,
attephstat INTEGER)
MOD00.P2022001.1820.CWFL_FULL.L1A_LAC2022-01-01T18:20:002022-01-01T18:24:50
N2022001_SST_OIV2AV_24h.nc/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/c/sstfile^
N2022001_SST_OIV2AV_24h.nc/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/c/icefile^
N202200200_MET_NCEPR2_6h.hdf/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/df/met3^
N202200200_MET_NCEPR2_6h.hdf/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/df/met2^
N202200118_MET_NCEPR2_6h.hdf/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/df/met1b
PM1EPHND.P2022001.1200.003/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/001eph1b
PM1ATTNR.P2022001.1800.003/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/001att1


And the files are in the correct locations for T2021.24 (AND the same directory as the the MET files that it found?):
ll /shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/002/N202200200_O3_AURAOMI_24h.hdf /shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/002/N202200200_O3_AURAOMI_24h.hdf /shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/001/N202200100_O3_AURAOMI_24h.hdf
-rw-rw-r-- 1 bmurch cms_optics 201286 Jan 3 08:28 /shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/001/N202200100_O3_AURAOMI_24h.hdf
-rw-rw-r-- 1 bmurch cms_optics 201286 Jan 5 00:02 /shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/002/N202200200_O3_AURAOMI_24h.hdf
-rw-rw-r-- 1 bmurch cms_optics 201286 Jan 5 00:02 /shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/002/N202200200_O3_AURAOMI_24h.hdf

Please advise.
Cheers,
Brock

Tags:

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 310
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1

Re: T2021.24 scripts not returning correctly

by OB SeaDAS - xuanyang02 » Sat Jan 22, 2022 11:31 am America/New_York

Brock,

Thanks for reporting the bug. It's fixed in V2022.0. However, getanc does not work with -s Option in V2022.0. See viewtopic.php?f=7&t=2843&sid=77cf7c94b1 ... 4aa6928029. It's fixed and will be in the next tag

Bing

oo_processing
Posts: 240
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

Re: T2021.24 scripts not returning correctly

by oo_processing » Mon Jan 24, 2022 3:47 pm America/New_York

So what is the proper method of doing this then?

I use both modis_atteph and getanc (specifically to get the proper files for processing from L0->L1B like your link error example "getanc -c -s 2022012040000 -e 2022012200000") and again in my l2gen station as well?

getanc --refreshDB -m aqua -s 2022024165002 -e 2022024165002 --verbose --ancdb=/shares/cms_optics/apps/seadas/ancillary_data.db.updated

And this is not great. We want to refresh the database with the -r or --refreshDB and we DO NOT want to d/l anything so use the --disable-download, but that doesn't refresh the DB at all even with the --refreshDB. Here is an example with an new database file (not what strings returns at the end):

$ getanc --refreshDB --disable-download -m aqua -s 2022024165002 -e 2022024165002 --verbose --ancdb=./ancillary_data_TEST.db
./ancillary_data_TEST.db

Input file: None
Sensor : aqua
Start time: 2022-01-24T16:50:02
End time : 2022-01-24T16:50:02

OBPG session started
Downloading 2022-01-24T16:50:02.anc.server ( 0.00 MBs)
[==================================================]
...Done
Found: /shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/024/N202202412_MET_NCEP_6h.hdf
Found: /shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/022/N202202200_SEAICE_NSIDC_24h.hdf
Found: /shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/023/N2022023_SST_OIV2AV_24h.nc
*** WARNING: No optimal OZONE files found.

Created '2022-01-24T16:50:02.anc' l2gen parameter text file:

att1=PM1ATTNR_NRT.A2022024.1645.061
att2=PM1ATTNR_NRT.A2022024.1650.061
eph1=PM1EPHND_NRT.A2022024.1645.061
eph2=PM1EPHND_NRT.A2022024.1650.061
icefile=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/022/N202202200_SEAICE_NSIDC_24h.hdf
met1=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/024/N202202412_MET_NCEP_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/024/N202202412_MET_NCEP_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/024/N202202412_MET_NCEP_6h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2022/023/N2022023_SST_OIV2AV_24h.nc


*** WARNING: The following ancillary data types were missing or are not optimal: OZONE
$ strings ./ancillary_data_TEST.db
SQLite format 3
Wtablesatancinfosatancinfo
CREATE TABLE satancinfo
(satid INTEGER ,
ancid INTEGER ,
optimal INTEGER,
FOREIGN KEY(satID) REFERENCES satfiles(satid),
FOREIGN KEY(ancID) REFERENCES ancfiles(ancid))
tableancfilesancfiles
CREATE TABLE ancfiles
(ancid INTEGER PRIMARY KEY,
filename TEXT ,
path TEXT ,
type TEXT)
%tablesatfilessatfiles
CREATE TABLE satfiles
(satid INTEGER PRIMARY KEY,
filename TEXT ,
starttime TEXT,
stoptime TEXT,
status INTEGER,
attephstat INTEGER)


The last 'stable' T2021.24 is has errors. The V tags are not 'stable'. So what to do?

This version with some changes worked but won't do newer satellites:
[bmurch@optics0 seadas]$ cat ./seadas-7.5.3/ocssw/bin/.git/FETCH_HEAD

fbc663585d82f2f013a45cc5d57926e295c4006a branch 'v7.5' of https://oceandata.sci.gsfc.nasa.gov/ocssw/bin-linux_64
fbc663585d82f2f013a45cc5d57926e295c4006a not-for-merge branch 'master' of https://oceandata.sci.gsfc.nasa.gov/ocssw/bin-linux_64
a9a30059455e02d53f2fb9c19395ca5771b151fe not-for-merge branch 'v7.0.2' of https://oceandata.sci.gsfc.nasa.gov/ocssw/bin-linux_64
a3a3cc6feb577aff95f39e106458aae5d8475ddc not-for-merge branch 'v7.1' of https://oceandata.sci.gsfc.nasa.gov/ocssw/bin-linux_64
7245b115949ba15a37fd4472f709aa5052449c9c not-for-merge branch 'v7.2' of https://oceandata.sci.gsfc.nasa.gov/ocssw/bin-linux_64
0e1a39a9bc1e47dbe42d75e9a0f59289eb9d996c not-for-merge branch 'v7.3' of https://oceandata.sci.gsfc.nasa.gov/ocssw/bin-linux_64
48b035ec8a8fca163a7193442c1e51a4efe7f03b not-for-merge branch 'v7.4' of https://oceandata.sci.gsfc.nasa.gov/ocssw/bin-linux_6

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 310
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1

Re: T2021.24 scripts not returning correctly

by OB SeaDAS - xuanyang02 » Mon Jan 24, 2022 4:06 pm America/New_York

Please use V2021.2 for now before V2022.1 or T2022.1 come out. Thanks for your patience

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 310
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1

Re: T2021.24 scripts not returning correctly

by OB SeaDAS - xuanyang02 » Tue Jan 25, 2022 4:54 pm America/New_York

T2022.1 is available now

Post Reply