modis_atteph returns anc files. Also, --disable-download yields no entries in --ancdb

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
bbbarnes
Posts: 31
Joined: Tue Feb 04, 2014 12:57 pm America/New_York
Answers: 0

modis_atteph returns anc files. Also, --disable-download yields no entries in --ancdb

by bbbarnes » Tue Oct 04, 2022 4:09 pm America/New_York

Apart from issues with ancillary naming convention (viewtopic.php?t=3543, viewtopic.php?p=12592), I'm seeing differing behavior in modis_atteph returns among the various tags/versions. For example, initializing our oldest local version of SeaDAS8.1 (from $OCSSWROOT/manifest.json, this appears to be T2021.11):

> source /shares/cms_optics/apps/seadas/seadas-8.1/ocssw/OCSSW_bash.env.cluster

Running modis_atteph correctly prints the att and eph files to stdout
> modis_atteph -s 2021204003001 -e 2021204003001 --ancdb=./anc_seadas8.1.db -m aqua
./anc_seadas8.1.db
att1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2021/204/PM1ATTNR.P2021204.0000.003
eph1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2021/203/PM1EPHND.P2021203.1200.003

Parsing the anc.db, only those two files are listed
> sqlite3 anc_seadas8.1.db "select filename from ancfiles"
PM1ATTNR.P2021204.0000.003
PM1EPHND.P2021203.1200.003

Everything looks good! The att/eph files are retuned in the stdout (and in an ASCII file), and are correctly stored in the anc.db file. However, running the same exact commands for T2021.24:

> source /shares/cms_optics/apps/seadas/ocssw/OCSSW_bash.env.cluster.T2021.24

> modis_atteph -s 2021204003001 -e 2021204003001 --ancdb=./anc_T2021.24.db -m aqua
./anc_T2021.24.db
att1=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2021/204/PM1ATTNR.P2021204.0000.003
eph1=/shares/cms_optics/apps/seadas/ocssw/T2021.24/var/anc/2021/203/PM1EPHND.P2021203.1200.003
icefile=20210723120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc
met1=GMAO_MERRA2.20210723T000000.MET.nc
met2=GMAO_MERRA2.20210723T010000.MET.nc
met3=GMAO_MERRA2.20210723T010000.MET.nc
ozone1=GMAO_MERRA2.20210723T000000.MET.nc
ozone2=GMAO_MERRA2.20210723T010000.MET.nc
ozone3=GMAO_MERRA2.20210723T010000.MET.nc
sstfile=20210723120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc


> sqlite3 anc_T2021.24.db "select filename from ancfiles"
PM1ATTNR.P2021204.0000.003
PM1EPHND.P2021203.1200.003
GMAO_MERRA2.20210723T000000.MET.nc
GMAO_MERRA2.20210723T010000.MET.nc
GMAO_MERRA2.20210723T010000.MET.nc
GMAO_MERRA2.20210723T000000.MET.nc
GMAO_MERRA2.20210723T010000.MET.nc
GMAO_MERRA2.20210723T010000.MET.nc
20210723120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc
20210723120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc


Note that anc files are returned in the stdout for a modis_atteph call, and are also stored in the anc.db. Newer versions (T2022.2, V2022.0, and R2022.14) have the stdout issue fixed, but the anc files are still returned in the anc.db:

> source /shares/cms_optics/apps/seadas/ocssw/OCSSW_bash.env.cluster.R2022.14

> modis_atteph -s 2021204003001 -e 2021204003001 --ancdb=./anc_R2022.14.db -m aqua
./anc_R2022.14.db
att1=/shares/cms_optics/apps/seadas/ocssw/R2022.14/var/anc/2021/204/PM1ATTNR.P2021204.0000.003
eph1=/shares/cms_optics/apps/seadas/ocssw/R2022.14/var/anc/2021/203/PM1EPHND.P2021203.1200.003

> sqlite3 anc_R2022.14.db "select filename from ancfiles"
PM1ATTNR.P2021204.0000.003
PM1EPHND.P2021203.1200.003
GMAO_MERRA2.20210723T000000.MET.nc
GMAO_MERRA2.20210723T010000.MET.nc
GMAO_MERRA2.20210723T010000.MET.nc
GMAO_MERRA2.20210723T000000.MET.nc
GMAO_MERRA2.20210723T010000.MET.nc
GMAO_MERRA2.20210723T010000.MET.nc
20210723120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc
20210723120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc


A related (but more longstanding) issue is that the --disable-download argument is incompatible with the --ancdb argument. When the former is activated, nothing is recorded in the specified anc.db file. For example:

> modis_atteph -s 2021204003001 -e 2021204003001 --ancdb=./anc_disable_download.db -m aqua --disable-download
./anc_disable_download.db
att1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2021/204/PM1ATTNR.P2021204.0000.003
eph1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2021/203/PM1EPHND.P2021203.1200.003

> sqlite3 anc_disable_download.db "select filename from ancfiles"
>

In the past, we've resolved this internally with modifications to anc_utils.py. However, others may have this issue, and modifying every version seems duplicative if it's something that can be resolved on OBPG's end...

Thanks in advance
-brian
Last edited by bbbarnes on Wed Oct 05, 2022 11:15 am America/New_York, edited 1 time in total.

Tags:

bbbarnes
Posts: 31
Joined: Tue Feb 04, 2014 12:57 pm America/New_York
Answers: 0

Re: modis_atteph returns ancdb files. Also, --disable-download yields no entries in --ancdb

by bbbarnes » Tue Oct 04, 2022 4:21 pm America/New_York

Seems Yuyuan and I are working on the same issue:
viewtopic.php?t=3543

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 405
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1

Re: modis_atteph returns anc files. Also, --disable-download yields no entries in --ancdb

by OB SeaDAS - xuanyang02 » Wed Oct 05, 2022 4:03 pm America/New_York

The issue "modis_atteph seems to put excess MET and SST records into sql database" is already fixed. Hopefully the fix will be in the next V tag.

Regarding the 2nd issue, --disable-download will neither download any anc/atteph files, nor insert the record in the database. It behaves as expected

bbbarnes
Posts: 31
Joined: Tue Feb 04, 2014 12:57 pm America/New_York
Answers: 0

Re: modis_atteph returns anc files. Also, --disable-download yields no entries in --ancdb

by bbbarnes » Wed Oct 05, 2022 4:47 pm America/New_York

Thanks.

Is there any option/configuration that disables download but updates the sql database? Reasoning: our workflow generally includes processing many L1 files - each requiring a call to getanc/modis_atteph. From previous experience, multiple sequential connections to NASA, or multiple requests for the same anc file within a certain timeframe (1 minute?) may result in our IP being blocked. Thus, our workaround approach is to (1) build out an anc.db (using getanc/modis_atteph, but without downloading anything) for all L1 files to be processed, (2) compile a list of missing anc files, then (3) download them all in a single curl call. Currently, that first part requires internal modification to anc_utils.py, which we have to implement for every new OCSSW tag we install. Is there a better way to be doing this?

-brian

Post Reply