Seadas T2022.2 modis_atteph and getanc issues

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
yuyuanxie
Posts: 23
Joined: Fri Mar 04, 2022 10:36 am America/New_York

Seadas T2022.2 modis_atteph and getanc issues

by yuyuanxie » Fri Mar 04, 2022 12:18 pm America/New_York

I was processing 3410 files of years between 2018 and 2021. Seadas 8.1 (T2021.12) is the version I'm currently working with. But for some reasons, I made a comparison between ancillary.db of T2022.2 and 8.1.

T2022.2 modis_atteph and getanc
(1) all cases returned sstfile record for icefile except those with "No optimal ICE files found".
(2) modis_atteph didn't return record and getanc didn't find optimal files (83/3410 chance), but 8.1 could
(3) return records different to those by 8.1 (41/3410 chance), eg., T2022.2 return 2-day old ozone1 file but next-day file by 8.1. I guess T2022.2 and 8.1 use different matching algorithms, but which one is better?

####################################################################################
[yuyuan@optics1 day10-21]$ getanc --disable-download --refreshDB -s 2018023141502 -e 2018023141502 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N2018023_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802312_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802318_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802318_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802300_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802300_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N201802300_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/023/N2018023_SST_OIV2AV_24h.nc
####################################################################################
[yuyuan@optics1 day10-21]$ modis_atteph --refreshDB --disable-download -s 2021187144000 -e 2021187144000 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db

[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2021187144000 -e 2021187144000 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
*** WARNING: No optimal MET files found.
*** WARNING: No optimal SST files found.
*** WARNING: No optimal ICE files found.
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2021/187/N202118700_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2021/188/N202118800_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2021/188/N202118800_O3_AURAOMI_24h.hdf


*** WARNING: The following ancillary data types were missing or are not optimal: MET SST Sea Ice
####################################################################################
[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2018365151502 -e 2018365151502 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N2018365_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N201836512_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N201836518_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N201836518_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/363/N201836300_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2019/001/N201900100_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2019/001/N201900100_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2018/365/N2018365_SST_OIV2AV_24h.nc

Tags:

OB WebDev - masoudso
Subject Matter Expert
Subject Matter Expert
Posts: 8
Joined: Tue Dec 22, 2020 2:11 pm America/New_York

Re: Seadas T2022.2 modis_atteph and getanc issues

by OB WebDev - masoudso » Tue Mar 08, 2022 11:28 am America/New_York

Thank you for your post. The issue has been resolved, and both versions of ancillary API should now return the same data.
Please let us know if you have any questions.

Masoud

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 380
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1

Re: Seadas T2022.2 modis_atteph and getanc issues

by OB SeaDAS - xuanyang02 » Tue Mar 08, 2022 1:32 pm America/New_York

And modis_atteph should work too --
$ modis_atteph --refreshDB --disable-download -s 2021187144000 -e 2021187144000 --ancdb=$(pwd)/test.db -m terra
Running debug binaries...
/Users/*/ocssw-T2022.2/bin
/Users/*/Scenes/MODIS_TERRA/run/test.db
att1=/Users/*/ocssw-T2022.2/var/anc/2021/187/AM1ATTN0.A2021187.1400.002
eph1=/Users/*/ocssw-T2022.2/var/anc/2021/187/AM1EPHN0.A2021187.1400.002

yuyuanxie
Posts: 23
Joined: Fri Mar 04, 2022 10:36 am America/New_York

Re: Seadas T2022.2 modis_atteph and getanc issues

by yuyuanxie » Wed Mar 09, 2022 5:04 pm America/New_York

OB WebDev - masoudso wrote:
> Thank you for your post. The issue has been resolved, and both versions of
> ancillary API should now return the same data.
> Please let us know if you have any questions.
>
> Masoud

Hello Masoud,
Thanks very much for your work. It's much better.
But these followings you may want to take a look:
(1) icefile N2020213_SST_OIV2AV_24h.nc vs N202021300_SEAICE_NSIDC_24h.hdf (clearly T2022.2 is still wrong)
(2) subtle difference in 24h ancfiles selection (ozone and sstfile, and possibly icefile, too). In 8.1, I guess it subtracted 12 hours from starttime then got YYYYDDD to match file names.
(3) 7.5.3 getanc.py threw an error "Error! could not establish a network connection. Check your network connection." (I don't know why since other two can connect to network)
-Yuyuan
######################################################################
The following examples show subtle differences for an 11AM datetime
######################################################################
[yuyuan@optics1 day10-21]$ which getanc
/shares/cms_optics/apps/seadas/ocssw/T2022.2/bin/getanc
[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2020213110002 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N2020213_SST_OIV2AV_24h.nc
met1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021306_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/214/N202021400_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/214/N202021400_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/ocssw/T2022.2/var/anc/2020/213/N2020213_SST_OIV2AV_24h.nc
######################################################################
[yuyuan@optics1 day10-21]$ which getanc
/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/bin/getanc
[yuyuan@optics1 day10-21]$ getanc --refreshDB --disable-download -s 2020213110002 --ancdb=$(pwd)/test.db -m terra
/work/y/yuyuan/proc_sat/P_Rico/day10-21/test.db
icefile=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/212/N202021200_SEAICE_NSIDC_24h.hdf
met1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021306_MET_NCEPR2_6h.hdf
met2=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
met3=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
ozone1=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/212/N202021200_O3_AURAOMI_24h.hdf
ozone2=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
ozone3=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
sstfile=/shares/cms_optics/apps/seadas/seadas-8.1/ocssw/var/anc/2020/212/N2020212_SST_OIV2AV_24h.nc
Last edited by yuyuanxie on Wed Mar 09, 2022 7:42 pm America/New_York, edited 4 times in total.

yuyuanxie
Posts: 23
Joined: Fri Mar 04, 2022 10:36 am America/New_York

Re: Seadas T2022.2 modis_atteph and getanc issues

by yuyuanxie » Wed Mar 09, 2022 6:09 pm America/New_York

Hello Masoud and Xuanyang,

This may relate to both of your corrections for no att/eph and no optimal anc files. It's corrected for August 2021, but July 2021 also needs correction. I still got no att/eph and no optimal anc files for dates between 2021-07-01 and 2021-07-31. Also, neither N202121312_MET_NCEPR2_6h.hdf nor N202121318_MET_NCEPR2_6h.hdf appeared in results (that's for August 1 2021). Sorry to bother you two again.

All the best,
-Yuyuan
Last edited by yuyuanxie on Wed Mar 09, 2022 7:18 pm America/New_York, edited 1 time in total.

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 380
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1

Re: Seadas T2022.2 modis_atteph and getanc issues

by OB SeaDAS - xuanyang02 » Thu Mar 10, 2022 5:31 pm America/New_York

Hi Yuyuan,

Thanks for reporting the getanc/modis_atteph issue with T2022.2. We are working on the new ancillary API, and will let you know once it returns the missing att/eph/icefile/sstfile files.

For those files the new API does return -- including the different icefiles or the ozone/sstfiles, they are not incorrect, for they are exactly what is done in production --
$ getanc --refreshDB --disable-download -s 2021210110002 -e 2021210110502 --ancdb=$(pwd)/test.db -m terra
Running debug binaries...
/Users/*/ocssw-T2022.2/bin
/Users/*/Scenes/MODIS_TERRA/run/test.db
icefile=/Users/*/ocssw-T2022.2/var/anc/2021/210/N2021210_SST_OIV2AV_24h.nc
met1=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121006_MET_NCEPR2_6h.hdf
met2=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121012_MET_NCEPR2_6h.hdf
met3=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121012_MET_NCEPR2_6h.hdf
ozone1=/Users/*/ocssw-T2022.2/var/anc/2021/210/N202121000_O3_AURAOMI_24h.hdf
ozone2=/Users/*/ocssw-T2022.2/var/anc/2021/211/N202121100_O3_AURAOMI_24h.hdf
ozone3=/Users/*/ocssw-T2022.2/var/anc/2021/211/N202121100_O3_AURAOMI_24h.hdf
sstfile=/Users/*/ocssw-T2022.2/var/anc/2021/210/N2021210_SST_OIV2AV_24h.nc

7.5.3 getanc.py should work -- Please try again
$ getanc.py --refreshDB --disable-download -s 2020213110002 --ancdb=$(pwd)/test.db -m terra
Running debug binaries...
/Applications/seadas-7.5.3/ocssw/bin
icefile=/Applications/seadas-7.5.3/ocssw/var/anc/2020/212/N202021200_SEAICE_NSIDC_24h.hdf
met1=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021306_MET_NCEPR2_6h.hdf
met2=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
met3=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021312_MET_NCEPR2_6h.hdf
ozone1=/Applications/seadas-7.5.3/ocssw/var/anc/2020/212/N202021200_O3_AURAOMI_24h.hdf
ozone2=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
ozone3=/Applications/seadas-7.5.3/ocssw/var/anc/2020/213/N202021300_O3_AURAOMI_24h.hdf
sstfile=/Applications/seadas-7.5.3/ocssw/var/anc/2020/212/N2020212_SST_OIV2AV_24h.nc

Bing
Last edited by OB SeaDAS - xuanyang02 on Wed Mar 16, 2022 10:35 am America/New_York, edited 2 times in total.

OB WebDev - dana.r.wilson
Subject Matter Expert
Subject Matter Expert
Posts: 66
Joined: Mon Apr 20, 2020 8:04 am America/New_York

Re: Seadas T2022.2 modis_atteph and getanc issues

by OB WebDev - dana.r.wilson » Fri Mar 11, 2022 8:35 am America/New_York

Yuyuan
The files have been restored.

yuyuanxie
Posts: 23
Joined: Fri Mar 04, 2022 10:36 am America/New_York

Re: Seadas T2022.2 modis_atteph and getanc issues

by yuyuanxie » Mon Mar 14, 2022 6:27 pm America/New_York

Sorry for the late response. It took me time to digest. But thank you, all the information you provided is useful.
And thank you for recovering files.

-Yuyuan

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 380
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1

Re: Seadas T2022.2 modis_atteph and getanc issues

by OB SeaDAS - xuanyang02 » Fri Apr 22, 2022 10:30 am America/New_York

Actually, l2gen has no problems taking an SST file as icefile.

melinfr
Posts: 8
Joined: Mon Aug 08, 2005 12:43 pm America/New_York

Re: Seadas T2022.2 modis_atteph and getanc issues

by melinfr » Tue Apr 26, 2022 5:18 am America/New_York

Dear support,
I am not sure I understand your answer. l2gen might not complain but if a test within l2gen expects a sea ice concentration, providing a SST value might not do a proper job. Could you please clarify ? Thanks a lot in advance,
Best, F.

Post Reply