Welcome to the Earthdata Forum! Here, the scientific user community and subject matter experts from NASA Distributed Active Archive Centers (DAACs), and other contributors, discuss research needs, data, and data applications.
by slwood » Fri Apr 27, 2018 2:27 am America/New_York
Hi Sean & co,
I've probably jumped the gun a bit here and this week updated to the HEAD of the master branch which I guess is v7.5 pre-release. (We're about to do some reprocessing so wanted to make sure we had the latest OCSSW installed (we were at v7.4).) Anyway it builds OK but I found some strange behaviour with the update_luts.py script.
Debugging some download failures that led to stack traces similar to those reported here https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=8483 (but which I now suspect were due to hitting your server yesterday just as you were updating some of the files) I was looking at LutUtils.get_luts(). At line 128 a regex is used apparently to filter out any files that contain version numbers so only non-versioned files get their times checked. But it doesn't work so all files get their times checked and then all are then checked again in the "for others" clause that follows...
The following output with a couple of extra print statements demonstrates (URL at line 123 of LutUtils, filepath at line 273 of JsonUtils) -- you can see every file (versioned or not) gets checked twice: $ update_luts.py -v aqua
Downloading files into /home/woodsl/var/ocssw/modis URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json filepath: /home/woodsl/var/ocssw/modis/leapsec.dat filepath: /home/woodsl/var/ocssw/modis/utcpole.dat filepath: /home/woodsl/var/ocssw/modis/leapsec.dat filepath: /home/woodsl/var/ocssw/modis/utcpole.dat ...no new files.
Downloading files into /home/woodsl/var/ocssw/modisa URL: https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json filepath: /home/woodsl/var/ocssw/modisa/?format=json filepath: /home/woodsl/var/ocssw/modisa/?format=json filepath: /home/woodsl/var/ocssw/modisa/?format=json filepath: /home/woodsl/var/ocssw/modisa/?format=json ...no new files.
by slwood » Tue May 01, 2018 1:14 am America/New_York
A while ago we encountered some problems with the LUT downloader where bad network connections causing retries exposed an issue in ProcUtils.httpdl() where the connection is closed and the transfer is retried with urlConn=None causing a new connection to be opened. The problem arises if reuseConn is set since the caller does not get the new connection and is left holding a reference to the original connection which is now closed. If the caller then tries to use the connection again (as happens a lot in lut_utils.py) Bad Things ensue...
I fixed this in our version by passing the new connection (urlConn) back out to the caller if reuseConn is set, returning (urlConn, status) rather than just status from httpdl().
I'll attach the required patches to this post in case you want to adopt them (there doesn't seem to be an option to attach a file until after I've hit 'post'??).
I have been having errors today in the L1B processing since I automatically update my LUTS each night (see the LUT version: xx.xxx.xx.xx below). On a different thread I mentioned to Sean that I work in a CentOS world, and I have a lot to do to compile everything. BUT THE NEW LUTS FAIL, and are pushed automatically. I know I can make it work with -d and -l directives like below, but this makes things more difficult.
Processing MODIS L1A file to L1B... MOD_PR02 version 6.1.12b built on Apr 26 2016, at 15:06:00 /optics1/software/seadas/seadas-7.4/ocssw/run/bin/linux_64/l1bgen_modist exit status: 1 ERROR: MODIS L1B processing failed. Please examine the LogStatus and LogUser files for more information.
HOWEVER When I use the older LUT it works: /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy> /optics1/software/seadas/seadas-7.4/ocssw/run/scripts/modis_L1B.py /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD00.A2018130.0330.CWFL.L1A_LAC /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD03.2018130.0330.CWFL.hdf -o /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD021KM.2018130.0330.CWFL.hdf -k /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD02HKM.2018130.0330.CWFL.hdf -q /tmp/S4P_ROI_H5_MODIST.CWFL.2018.130_version_0.03_Yz_dkNvy/MOD02QKM.2018130.0330.CWFL.hdf -v -d /optics1/software/seadas/seadas-7.4/ocssw/run/var/modist/cal/OPER -l xx.xxx.xx.xx
Processing MODIS L1A file to L1B... MOD_PR02 version 6.1.12b built on Apr 26 2016, at 15:06:00 scan: 0 out of 28 Thu May 10 10:25:09 2018 scan: 10 out of 28 Thu May 10 10:25:09 2018 scan: 20 out of 28 Thu May 10 10:25:09 2018 /optics1/software/seadas/seadas-7.4/ocssw/run/bin/linux_64/l1bgen_modist exit status: 0 MODIS L1B processing complete.
So, why does the lut update not use the correct LUT?
You are pulling new LUTs; we are not pushing them.
If you must continue using v7.4 code, you have two choices: * Stop pulling the modist LUTs, and put the last MOD02_*_LUTs.V6.1.*.hdf files back into var/modist/cal/OPER * Specify the -d and -l directives.
by zhigang » Fri May 11, 2018 1:36 am America/New_York
I tried again today, it still failed. And I searched the prolem in google, it caused by the privoxy. In China, I cannot directly access the ocean color and its service, now I used the privoxy to access oceancolor website. The privoxy cannot support the intercepting encrypted connections (HTTPS). Therefore, I just downloaded the luts manually from the oceandata (https://oceandata.sci.gsfc.nasa.gov/).