possible issue with v7.5 lut updater?

Please enter here to ask a question about any NASA Science related topics!
slwood
Posts: 9
Joined: Thu May 03, 2007 6:41 pm America/New_York

possible issue with v7.5 lut updater?

by slwood » Fri Apr 27, 2018 2:27 am America/New_York

Hi Sean & co,

I've probably jumped the gun a bit here and this week updated to the HEAD of the master branch which I guess is v7.5 pre-release.  (We're about to do some reprocessing so wanted to make sure we had the latest OCSSW installed (we were at v7.4).)  Anyway it builds OK but I found some strange behaviour with the update_luts.py script.

Debugging some download failures that led to stack traces similar to those reported here https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=8483 (but which I now suspect were due to hitting your server yesterday just as you were updating some of the files) I was looking at LutUtils.get_luts(). At line 128 a regex is used apparently to filter out any files that contain version numbers so only non-versioned files get their times checked.  But it doesn't work so all files get their times checked and then all are then checked again in the "for others" clause that follows...

The following output with a couple of extra print statements demonstrates (URL at line 123 of LutUtils, filepath at line 273 of JsonUtils) -- you can see every file (versioned or not) gets checked twice:

$ update_luts.py -v aqua

Downloading files into /home/woodsl/var/ocssw/modis
URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json
filepath: /home/woodsl/var/ocssw/modis/leapsec.dat
filepath: /home/woodsl/var/ocssw/modis/utcpole.dat
filepath: /home/woodsl/var/ocssw/modis/leapsec.dat
filepath: /home/woodsl/var/ocssw/modis/utcpole.dat
...no new files.

Downloading files into /home/woodsl/var/ocssw/modisa
URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json
filepath: /home/woodsl/var/ocssw/modisa/?format=json
filepath: /home/woodsl/var/ocssw/modisa/?format=json
filepath: /home/woodsl/var/ocssw/modisa/?format=json
filepath: /home/woodsl/var/ocssw/modisa/?format=json
...no new files.

Downloading files into /home/woodsl/var/ocssw/modisa/cal/OPER
URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_Emissive_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_QA_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_Reflective_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_Emissive_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_QA_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_Reflective_LUTs.V6.2.1.10_OC_v1.10.hdf
...no new files.

Downloading files into /home/woodsl/var/ocssw/modisa/xcal/OPER
URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_1240.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_1640.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_2130.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_412.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_443.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_469.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_488.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_531.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_547.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_555.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_645.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_667.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_678.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_748.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_859.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_869.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_1240.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_1640.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_2130.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_412.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_443.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_469.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_488.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_531.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_547.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_555.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_645.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_667.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_678.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_748.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_859.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_869.hdf
...no new files.


I guess it's not really doing any harm -- but it's not achieving the intended goal either... just thought I'd pass it up...

cheers
Simon

Tags:

gfireman
Posts: 59
Joined: Thu Jan 07, 2010 2:59 pm America/New_York

possible issue with v7.5 lut updater?

by gfireman » Fri Apr 27, 2018 11:53 am America/New_York

Hi, Simon -

I'm the "& co" who wrote that bit of code.  It used to work!  I'll look into it.

Thanks for the heads-up!
Gwyn

slwood
Posts: 9
Joined: Thu May 03, 2007 6:41 pm America/New_York

possible issue with v7.5 lut updater?

by slwood » Tue May 01, 2018 1:14 am America/New_York

Hi Gwyn,

A while ago we encountered some problems with the LUT downloader where bad network connections causing retries exposed an issue in ProcUtils.httpdl() where the connection is closed and the transfer is retried with urlConn=None causing a new connection to be opened.  The problem arises if reuseConn is set since the caller does not get the new connection and is left holding a reference to the original connection which is now closed.  If the caller then tries to use the connection again (as happens a lot in lut_utils.py) Bad Things ensue...

I fixed this in our version by passing the new connection (urlConn) back out to the caller if reuseConn is set, returning (urlConn, status) rather than just status from httpdl().

I'll attach the required patches to this post in case you want to adopt them (there doesn't seem to be an option to attach a file until after I've hit 'post'??).

regards
Simon
attachment 1

zhigang
Posts: 69
Joined: Tue Nov 10, 2020 8:03 pm America/New_York

possible issue with v7.5 lut updater?

by zhigang » Thu May 10, 2018 3:19 am America/New_York

Dear Gwyn?

Today, I updated luts of aqua in v7.5 branch, but a exception was thrown. The detail was:
[optics@localhost seadas-7.4]$ update_luts.py aqua --verbose

Downloading files into /home/optics/seadas-7.4/ocssw/var/modis
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Invalid header received from client
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Invalid header received from client
...no new files.

Downloading files into /home/optics/seadas-7.4/ocssw/var/modisa
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json: 400  Invalid header received from client
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json: 400  Invalid header received from client
...no new files.

Downloading files into /home/optics/seadas-7.4/ocssw/var/modisa/cal/OPER
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json: 400  Invalid header received from client
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json: 400  Invalid header received from client
...no new files.

Downloading files into /home/optics/seadas-7.4/ocssw/var/modisa/xcal/OPER
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json: 400  Invalid header received from client
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json: 400  Invalid header received from client
...no new files.

I never saw this type of problem before.

Best Regards,
Zhigang

gfireman
Posts: 59
Joined: Thu Jan 07, 2010 2:59 pm America/New_York

possible issue with v7.5 lut updater?

by gfireman » Thu May 10, 2018 10:37 am America/New_York

Could you please try again?  You might have been caught in between server updates.

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

possible issue with v7.5 lut updater?

by oo_processing » Thu May 10, 2018 11:12 am America/New_York

I have been having errors today in the L1B processing since I automatically update my LUTS each night (see the LUT version: xx.xxx.xx.xx below). On a different thread I mentioned to Sean that I work in a CentOS world, and I have a lot to do to compile everything.
BUT THE NEW LUTS FAIL, and are pushed automatically. I know I can make it work with -d and -l directives like below, but this makes things more difficult.

eg:
/optics1/software/seadas/seadas-7.4/ocssw/run/scripts/modis_L1B.py /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD00.A2018130.0330.CWFL.L1A_LAC /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD03.2018130.0330.CWFL.hdf -o /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD021KM.2018130.0330.CWFL.hdf -k /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkN vy/MOD02HKM.2018130.0330.CWFL.hdf -q /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD02QKM.2018130.0330.CWFL.hdf -v                                                                                                           

LUT directory: /optics1/software/seadas/seadas-7.4/ocssw/run/var/modist/cal/OPER
LUT version: xx.xxx.xx.xx                                                          
Reflective LUT: MOD02_Reflective_LUTs.Vxx.xxx.xx.xx.hdf                            
Emissive LUT: MOD02_Emissive_LUTs.Vxx.xxx.xx.xx.hdf                                
QA LUT: MOD02_QA_LUTs.Vxx.xxx.xx.xx.hdf                                            

Processing MODIS L1A file to L1B...
MOD_PR02 version 6.1.12b built on Apr 26 2016, at 15:06:00
/optics1/software/seadas/seadas-7.4/ocssw/run/bin/linux_64/l1bgen_modist exit status: 1
ERROR: MODIS L1B processing failed.                                                   
Please examine the LogStatus and LogUser files for more information.

HOWEVER
When I use the older LUT it works:
/tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy> /optics1/software/seadas/seadas-7.4/ocssw/run/scripts/modis_L1B.py /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD00.A2018130.0330.CWFL.L1A_LAC /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD03.2018130.0330.CWFL.hdf -o /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD021KM.2018130.0330.CWFL.hdf -k /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD02HKM.2018130.0330.CWFL.hdf -q /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD02QKM.2018130.0330.CWFL.hdf -v -d /optics1/software/seadas/seadas-7.4/ocssw/run/var/modist/cal/OPER -l xx.xxx.xx.xx

LUT directory: /optics1/software/seadas/seadas-7.4/ocssw/run/var/modist/cal/OPER
LUT version: xx.xxx.xx.xx
Reflective LUT: MOD02_Reflective_LUTs.Vxx.xxx.xx.xx.hdf
Emissive LUT: MOD02_Emissive_LUTs.Vxx.xxx.xx.xx.hdf
QA LUT: MOD02_QA_LUTs.Vxx.xxx.xx.xx.hdf

Processing MODIS L1A file to L1B...
MOD_PR02 version 6.1.12b built on Apr 26 2016, at 15:06:00
scan: 0 out of 28 Thu May 10 10:25:09 2018
scan: 10 out of 28 Thu May 10 10:25:09 2018
scan: 20 out of 28 Thu May 10 10:25:09 2018
/optics1/software/seadas/seadas-7.4/ocssw/run/bin/linux_64/l1bgen_modist exit status: 0
MODIS L1B processing complete.

So, why does the lut update not use the correct LUT?

Brock

gfireman
Posts: 59
Joined: Thu Jan 07, 2010 2:59 pm America/New_York

possible issue with v7.5 lut updater?

by gfireman » Thu May 10, 2018 11:29 am America/New_York

Brock -

The Terra calibration LUT format has changed; the new LUTs are intended to work with SeaDAS 7.5 code.
See https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=36918

You are pulling new LUTs; we are not pushing them.

If you must continue using v7.4 code, you have two choices:
* Stop pulling the modist LUTs, and put the last MOD02_*_LUTs.V6.1.*.hdf files back into var/modist/cal/OPER
* Specify the -d and -l directives.

- Gwyn

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1219
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

possible issue with v7.5 lut updater?

by OB.DAAC - SeanBailey » Thu May 10, 2018 2:31 pm America/New_York

The format of the Aqua LUT did not change, it was just the latest regular monthly update...and it works for me...

Sean

zhigang
Posts: 69
Joined: Tue Nov 10, 2020 8:03 pm America/New_York

possible issue with v7.5 lut updater?

by zhigang » Fri May 11, 2018 1:36 am America/New_York

Dear Gwyn,

I tried again today, it still failed. And I searched the prolem in google, it caused by the privoxy. In China, I cannot directly access the ocean color and its service, now I used the privoxy to access oceancolor website. The privoxy cannot support the intercepting encrypted connections (HTTPS). Therefore, I just downloaded the luts manually from the oceandata (https://oceandata.sci.gsfc.nasa.gov/).

Regards,
Zhigang

gfireman
Posts: 59
Joined: Thu Jan 07, 2010 2:59 pm America/New_York

possible issue with v7.5 lut updater?

by gfireman » Fri May 11, 2018 9:55 am America/New_York

I'm glad you found a workaround.  You'll likely run into this problem again; most sites in the U.S. are moving to using https.

Post Reply