Page 1 of 2

Seadas T2022.2 modis_atteph and getanc issues

Posted: Fri Mar 04, 2022 12:18 pm America/New_York
by yuyuanxie
I was processing 3410 files of years between 2018 and 2021. Seadas 8.1 (T2021.12) is the version I'm currently working with. But for some reasons, I made a comparison between ancillary.db of T2022.2 and 8.1.

T2022.2 modis_atteph and getanc
(1) all cases returned sstfile record for icefile except those with "No optimal ICE files found".
(2) modis_atteph didn't return record and getanc didn't find optimal files (83/3410 chance), but 8.1 could
(3) return records different to those by 8.1 (41/3410 chance), eg., T2022.2 return 2-day old ozone1 file but next-day file by 8.1. I guess T2022.2 and 8.1 use different matching algorithms, but which one is better?

####################################################################################
[yuyuan@optics1 day10-21]$ getanc --disable-download --refreshDB -s 2018023141502 -e 2018023141502 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N2018023_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802312_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802318_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802318_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802300_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802300_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802300_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N2018023_SST_OIV2AV_24h.nc
####################################################################################
[yuyuan@optics1 day10-21]$ modis_atteph --refreshDB --disable-download -s 2021187144000 -e 2021187144000 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db

[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2021187144000 -e 2021187144000 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
*** WARNING: No optimal MET files found.
*** WARNING: No optimal SST files found.
*** WARNING: No optimal ICE files found.
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2021/187/N202118700_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2021/188/N202118800_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2021/188/N202118800_O3_AURAOMI_24h.hdf


*** WARNING: The following ancillary data types were missing or are not optimal: MET SST Sea Ice
####################################################################################
[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2018365151502 -e 2018365151502 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N2018365_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N201836512_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N201836518_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N201836518_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/363/N201836300_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2019/001/N201900100_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2019/001/N201900100_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N2018365_SST_OIV2AV_24h.nc

Re: Seadas T2022.2 modis_atteph and getanc issues

Posted: Tue Mar 08, 2022 11:28 am America/New_York
by OB WebDev - masoudso
Thank you for your post. The issue has been resolved, and both versions of ancillary API should now return the same data.
Please let us know if you have any questions.

Masoud

Re: Seadas T2022.2 modis_atteph and getanc issues

Posted: Tue Mar 08, 2022 1:32 pm America/New_York
by OB SeaDAS - xuanyang02
And modis_atteph should work too --
$ modis_atteph --refreshDB --disable-download -s 2021187144000 -e 2021187144000 --ancdb=$(pwd)/test.db -m terra
Running debug binaries...
/Users/*/ocssw-T2022.2/bin
/Users/*/Scenes/MODIS_TERRA/run/test.db
att1=/Users/*/ocssw-T2022.2/var/anc/2021/187/AM1ATTN0.A2021187.1400.002
eph1=/Users/*/ocssw-T2022.2/var/anc/2021/187/AM1EPHN0.A2021187.1400.002

Re: Seadas T2022.2 modis_atteph and getanc issues

Posted: Wed Mar 09, 2022 5:04 pm America/New_York
by yuyuanxie
OB WebDev - masoudso wrote:
> Thank you for your post. The issue has been resolved, and both versions of
> ancillary API should now return the same data.
> Please let us know if you have any questions.
>
> Masoud

Hello Masoud,
Thanks very much for your work. It's much better.
But these followings you may want to take a look:
(1) icefile N2020213_SST_OIV2AV_24h.nc vs N202021300_SEAICE_NSIDC_24h.hdf (clearly T2022.2 is still wrong)
(2) subtle difference in 24h ancfiles selection (ozone and sstfile, and possibly icefile, too). In 8.1, I guess it subtracted 12 hours from starttime then got YYYYDDD to match file names.
(3) 7.5.3 getanc.py threw an error "Error! could not establish a network connection. Check your network connection." (I don't know why since other two can connect to network)
-Yuyuan
######################################################################
The following examples show subtle differences for an 11AM datetime
######################################################################
[yuyuan@optics1 day10-21]$ which getanc
/shares/cms_optics/apps/seadas/ocssw/T2022.2/bin/getanc
[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2020213110002 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N2020213_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021306_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/214/N202021400_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/214/N202021400_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N2020213_SST_OIV2AV_24h.nc
######################################################################
[yuyuan@optics1 day10-21]$ which getanc
/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/bin/getanc
[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2020213110002 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/212/N202021200_SEAICE_NSIDC_24h.hdf
met1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021306_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/212/N202021200_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/212/N2020212_SST_OIV2AV_24h.nc

Re: Seadas T2022.2 modis_atteph and getanc issues

Posted: Wed Mar 09, 2022 6:09 pm America/New_York
by yuyuanxie
Hello Masoud and Xuanyang,

This may relate to both of your corrections for no att/eph and no optimal anc files. It's corrected for August 2021, but July 2021 also needs correction. I still got no att/eph and no optimal anc files for dates between 2021-07-01 and 2021-07-31. Also, neither N202121312_MET_NCEPR2_6h.hdf nor N202121318_MET_NCEPR2_6h.hdf appeared in results (that's for August 1 2021). Sorry to bother you two again.

All the best,
-Yuyuan

Re: Seadas T2022.2 modis_atteph and getanc issues

Posted: Thu Mar 10, 2022 5:31 pm America/New_York
by OB SeaDAS - xuanyang02
Hi Yuyuan,

Thanks for reporting the getanc/modis_atteph issue with T2022.2. We are working on the new ancillary API, and will let you know once it returns the missing att/eph/icefile/sstfile files.

For those files the new API does return -- including the different icefiles or the ozone/sstfiles, they are not incorrect, for they are exactly what is done in production --
$ getanc --refreshDB --disable-download -s 2021210110002 -e 2021210110502 --ancdb=$(pwd)/test.db -m terra
Running debug binaries...
/Users/*/ocssw-T2022.2/bin
/Users/*/Scenes/MODIS_TERRA/run/test.db
icefile=/Users/*/ocssw-T2022.2/var/anc/2021/210/N2021210_SST_OIV2AV_24h.nc
met1=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121006_MET_NCEPR2_6h.hdf
met2=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121012_MET_NCEPR2_6h.hdf
met3=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121012_MET_NCEPR2_6h.hdf
ozone1=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121000_O3_AURAOMI_24h.hdf
ozone2=/Users/*/ocssw-T2022.2/var/anc/2021/211/N202121100_O3_AURAOMI_24h.hdf
ozone3=/Users/*/ocssw-T2022.2/var/anc/2021/211/N202121100_O3_AURAOMI_24h.hdf
sstfile=/Users/*/ocssw-T2022.2/var/anc/2021/210/N2021210_SST_OIV2AV_24h.nc

7.5.3 getanc.py should work -- Please try again
$ getanc.py --refreshDB --disable-download -s 2020213110002 --ancdb=$(pwd)/test.db -m terra
Running debug binaries...
/Applications/seadas-7.5.3/ocssw/bin
icefile=/Applications/seadas-7.5.3/ocssw/var/anc/2020/212/N202021200_SEAICE_NSIDC_24h.hdf
met1=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021306_MET_NCEPR2_6h.hdf
met2=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
met3=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
ozone1=/Applications/seadas-7.5.3/ocssw/var/anc/2020/212/N202021200_O3_AURAOMI_24h.hdf
ozone2=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
ozone3=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
sstfile=/Applications/seadas-7.5.3/ocssw/var/anc/2020/212/N2020212_SST_OIV2AV_24h.nc

Bing

Re: Seadas T2022.2 modis_atteph and getanc issues

Posted: Fri Mar 11, 2022 8:35 am America/New_York
by OB WebDev - dana.r.wilson
Yuyuan
The files have been restored.

Re: Seadas T2022.2 modis_atteph and getanc issues

Posted: Mon Mar 14, 2022 6:27 pm America/New_York
by yuyuanxie
Sorry for the late response. It took me time to digest. But thank you, all the information you provided is useful.
And thank you for recovering files.

-Yuyuan

Re: Seadas T2022.2 modis_atteph and getanc issues

Posted: Fri Apr 22, 2022 10:30 am America/New_York
by OB SeaDAS - xuanyang02
Actually, l2gen has no problems taking an SST file as icefile.

Re: Seadas T2022.2 modis_atteph and getanc issues

Posted: Tue Apr 26, 2022 5:18 am America/New_York
by melinfr
Dear support,
I am not sure I understand your answer. l2gen might not complain but if a test within l2gen expects a sea ice concentration, providing a SST value might not do a proper job. Could you please clarify ? Thanks a lot in advance,
Best, F.