Seadas T2022.2 modis_atteph and getanc issues

[yuyuanxie]

I was processing 3410 files of years between 2018 and 2021. Seadas 8.1 (T2021.12) is the version I'm currently working with. But for some reasons, I made a comparison between ancillary.db of T2022.2 and 8.1.

T2022.2 modis_atteph and getanc
(1) all cases returned sstfile record for icefile except those with "No optimal ICE files found".
(2) modis_atteph didn't return record and getanc didn't find optimal files (83/3410 chance), but 8.1 could
(3) return records different to those by 8.1 (41/3410 chance), eg., T2022.2 return 2-day old ozone1 file but next-day file by 8.1. I guess T2022.2 and 8.1 use different matching algorithms, but which one is better?

####################################################################################
[yuyuan@optics1 day10-21]$ getanc --disable-download --refreshDB -s 2018023141502 -e 2018023141502 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N2018023_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802312_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802318_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802318_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802300_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802300_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802300_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N2018023_SST_OIV2AV_24h.nc
####################################################################################
[yuyuan@optics1 day10-21]$ modis_atteph --refreshDB --disable-download -s 2021187144000 -e 2021187144000 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db

[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2021187144000 -e 2021187144000 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
*** WARNING: No optimal MET files found.
*** WARNING: No optimal SST files found.
*** WARNING: No optimal ICE files found.
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2021/187/N202118700_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2021/188/N202118800_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2021/188/N202118800_O3_AURAOMI_24h.hdf


*** WARNING: The following ancillary data types were missing or are not optimal: MET SST Sea Ice
####################################################################################
[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2018365151502 -e 2018365151502 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N2018365_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N201836512_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N201836518_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N201836518_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/363/N201836300_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2019/001/N201900100_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2019/001/N201900100_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N2018365_SST_OIV2AV_24h.nc

[masoudso]

Thank you for your post. The issue has been resolved, and both versions of ancillary API should now return the same data.
Please let us know if you have any questions.

Masoud

[OB SeaDAS - xuanyang02]

And modis_atteph should work too --
$ modis_atteph --refreshDB --disable-download -s 2021187144000 -e 2021187144000 --ancdb=$(pwd)/test.db -m terra
Running debug binaries...
/Users/*/ocssw-T2022.2/bin
/Users/*/Scenes/MODIS_TERRA/run/test.db
att1=/Users/*/ocssw-T2022.2/var/anc/2021/187/AM1ATTN0.A2021187.1400.002
eph1=/Users/*/ocssw-T2022.2/var/anc/2021/187/AM1EPHN0.A2021187.1400.002

[yuyuanxie]

OB WebDev - masoudso wrote:
> Thank you for your post. The issue has been resolved, and both versions of
> ancillary API should now return the same data.
> Please let us know if you have any questions.
>
> Masoud

Hello Masoud,
Thanks very much for your work. It's much better.
But these followings you may want to take a look:
(1) icefile N2020213_SST_OIV2AV_24h.nc vs N202021300_SEAICE_NSIDC_24h.hdf (clearly T2022.2 is still wrong)
(2) subtle difference in 24h ancfiles selection (ozone and sstfile, and possibly icefile, too). In 8.1, I guess it subtracted 12 hours from starttime then got YYYYDDD to match file names.
(3) 7.5.3 getanc.py threw an error "Error! could not establish a network connection. Check your network connection." (I don't know why since other two can connect to network)
-Yuyuan
######################################################################
The following examples show subtle differences for an 11AM datetime
######################################################################
[yuyuan@optics1 day10-21]$ which getanc
/shares/cms_optics/apps/seadas/ocssw/T2022.2/bin/getanc
[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2020213110002 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N2020213_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021306_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/214/N202021400_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/214/N202021400_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N2020213_SST_OIV2AV_24h.nc
######################################################################
[yuyuan@optics1 day10-21]$ which getanc
/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/bin/getanc
[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2020213110002 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/212/N202021200_SEAICE_NSIDC_24h.hdf
met1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021306_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/212/N202021200_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/212/N2020212_SST_OIV2AV_24h.nc

[yuyuanxie]

Hello Masoud and Xuanyang,

This may relate to both of your corrections for no att/eph and no optimal anc files. It's corrected for August 2021, but July 2021 also needs correction. I still got no att/eph and no optimal anc files for dates between 2021-07-01 and 2021-07-31. Also, neither N202121312_MET_NCEPR2_6h.hdf nor N202121318_MET_NCEPR2_6h.hdf appeared in results (that's for August 1 2021). Sorry to bother you two again.

All the best,
-Yuyuan

[OB SeaDAS - xuanyang02]

Hi Yuyuan,

Thanks for reporting the getanc/modis_atteph issue with T2022.2. We are working on the new ancillary API, and will let you know once it returns the missing att/eph/icefile/sstfile files.

For those files the new API does return -- including the different icefiles or the ozone/sstfiles, they are not incorrect, for they are exactly what is done in production --
$ getanc --refreshDB --disable-download -s 2021210110002 -e 2021210110502 --ancdb=$(pwd)/test.db -m terra
Running debug binaries...
/Users/*/ocssw-T2022.2/bin
/Users/*/Scenes/MODIS_TERRA/run/test.db
icefile=/Users/*/ocssw-T2022.2/var/anc/2021/210/N2021210_SST_OIV2AV_24h.nc
met1=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121006_MET_NCEPR2_6h.hdf
met2=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121012_MET_NCEPR2_6h.hdf
met3=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121012_MET_NCEPR2_6h.hdf
ozone1=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121000_O3_AURAOMI_24h.hdf
ozone2=/Users/*/ocssw-T2022.2/var/anc/2021/211/N202121100_O3_AURAOMI_24h.hdf
ozone3=/Users/*/ocssw-T2022.2/var/anc/2021/211/N202121100_O3_AURAOMI_24h.hdf
sstfile=/Users/*/ocssw-T2022.2/var/anc/2021/210/N2021210_SST_OIV2AV_24h.nc

7.5.3 getanc.py should work -- Please try again
$ getanc.py --refreshDB --disable-download -s 2020213110002 --ancdb=$(pwd)/test.db -m terra
Running debug binaries...
/Applications/seadas-7.5.3/ocssw/bin
icefile=/Applications/seadas-7.5.3/ocssw/var/anc/2020/212/N202021200_SEAICE_NSIDC_24h.hdf
met1=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021306_MET_NCEPR2_6h.hdf
met2=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
met3=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
ozone1=/Applications/seadas-7.5.3/ocssw/var/anc/2020/212/N202021200_O3_AURAOMI_24h.hdf
ozone2=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
ozone3=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
sstfile=/Applications/seadas-7.5.3/ocssw/var/anc/2020/212/N2020212_SST_OIV2AV_24h.nc

Bing

[dana.r.wilson]

Yuyuan
The files have been restored.

[yuyuanxie]

Sorry for the late response. It took me time to digest. But thank you, all the information you provided is useful.
And thank you for recovering files.

-Yuyuan

[OB SeaDAS - xuanyang02]

Actually, l2gen has no problems taking an SST file as icefile.

[melinfr]

Dear support,
I am not sure I understand your answer. l2gen might not complain but if a test within l2gen expects a sea ice concentration, providing a SST value might not do a proper job. Could you please clarify ? Thanks a lot in advance,
Best, F.