possible issue with v7.5 lut updater?
Posted: Fri Apr 27, 2018 2:27 am America/New_York
Hi Sean & co,
I've probably jumped the gun a bit here and this week updated to the HEAD of the master branch which I guess is v7.5 pre-release. (We're about to do some reprocessing so wanted to make sure we had the latest OCSSW installed (we were at v7.4).) Anyway it builds OK but I found some strange behaviour with the
Debugging some download failures that led to stack traces similar to those reported here https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=8483 (but which I now suspect were due to hitting your server yesterday just as you were updating some of the files) I was looking at LutUtils.get_luts(). At line 128 a regex is used apparently to filter out any files that contain version numbers so only non-versioned files get their times checked. But it doesn't work so all files get their times checked and then all are then checked again in the "for others" clause that follows...
The following output with a couple of extra print statements demonstrates (URL at line 123 of LutUtils, filepath at line 273 of JsonUtils) -- you can see every file (versioned or not) gets checked twice:
I guess it's not really doing any harm -- but it's not achieving the intended goal either... just thought I'd pass it up...
cheers
Simon
I've probably jumped the gun a bit here and this week updated to the HEAD of the master branch which I guess is v7.5 pre-release. (We're about to do some reprocessing so wanted to make sure we had the latest OCSSW installed (we were at v7.4).) Anyway it builds OK but I found some strange behaviour with the
update_luts.py
script.Debugging some download failures that led to stack traces similar to those reported here https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=8483 (but which I now suspect were due to hitting your server yesterday just as you were updating some of the files) I was looking at LutUtils.get_luts(). At line 128 a regex is used apparently to filter out any files that contain version numbers so only non-versioned files get their times checked. But it doesn't work so all files get their times checked and then all are then checked again in the "for others" clause that follows...
The following output with a couple of extra print statements demonstrates (URL at line 123 of LutUtils, filepath at line 273 of JsonUtils) -- you can see every file (versioned or not) gets checked twice:
$ update_luts.py -v aqua
Downloading files into /home/woodsl/var/ocssw/modis
URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json
filepath: /home/woodsl/var/ocssw/modis/leapsec.dat
filepath: /home/woodsl/var/ocssw/modis/utcpole.dat
filepath: /home/woodsl/var/ocssw/modis/leapsec.dat
filepath: /home/woodsl/var/ocssw/modis/utcpole.dat
...no new files.
Downloading files into /home/woodsl/var/ocssw/modisa
URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json
filepath: /home/woodsl/var/ocssw/modisa/?format=json
filepath: /home/woodsl/var/ocssw/modisa/?format=json
filepath: /home/woodsl/var/ocssw/modisa/?format=json
filepath: /home/woodsl/var/ocssw/modisa/?format=json
...no new files.
Downloading files into /home/woodsl/var/ocssw/modisa/cal/OPER
URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_Emissive_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_QA_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_Reflective_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_Emissive_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_QA_LUTs.V6.2.1.10_OC_v1.10.hdf
filepath: /home/woodsl/var/ocssw/modisa/cal/OPER/MYD02_Reflective_LUTs.V6.2.1.10_OC_v1.10.hdf
...no new files.
Downloading files into /home/woodsl/var/ocssw/modisa/xcal/OPER
URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_1240.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_1640.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_2130.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_412.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_443.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_469.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_488.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_531.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_547.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_555.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_645.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_667.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_678.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_748.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_859.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_869.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_1240.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_1640.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_2130.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_412.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_443.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_469.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_488.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_531.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_547.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_555.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_645.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_667.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_678.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_748.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_859.hdf
filepath: /home/woodsl/var/ocssw/modisa/xcal/OPER/xcal_modisa_axc_oc_v1.10d_869.hdf
...no new files.
I guess it's not really doing any harm -- but it's not achieving the intended goal either... just thought I'd pass it up...
cheers
Simon