Update the ocssw processors failed

[OB.DAAC - SeanBailey]

I have asked sysadmins for some brainstorming on this.  There may or may not be a solution or workaround to be found, but we'll try.
Unfortunately, if we want to keep our servers available we cannot use older ciphers.  To do so puts us in jeopardy of having the lights
turned off on us.

Sean

[gnwiii]

NASA is not the only organization moving to https or beefing up crypto protocols for https.  Many sites will be adopting the Mozilla Modern recommendations so if there are problems with NASA now there will be problems with others sites in the future.   It would be helpful to have some guidance for end user workarounds, e.g., for sites such as mine where git is failing, can we install a recent linux distro on an (older, unloved, and unwanted) PC, run install_ocssw.py there, and then use rsync to update the .git directories on the much loved trusty [pun intended] LTS machines?

[woodbri]

You might also add a note to the gnults bug mentioned above that this is impacting all .gov sites and hopefully they will up the priority on this issue and fix it in TLS where it should probably be dealt with. Meanwhile, sounds like I need to figure out a work around. Thanks all! I'll watch this thread for any updates.

[woodbri]

Could you create a static v7.2 and a v7.3 tarball of the git respository that we can download then build it manually? It seems that the only problem we have to in installing the software. Once we get it installed we should be good if we do not need to use git https access.

Would this work?

[swathy]

Hi,

I was also trying to update SeaDAS manually since its not happening with GUI. Its not able to communicate with the server after when it reaches towards the end.

rslab@rslab-pc:~/softwares$ sudo ./install_ocssw.py --install-dir=/home/rslab/seadas-7.3.2/ocssw --git-branch=v7.3 --aqua --seawifs --meris
Installing bundles.sha256sum (1 of 16)
--2017-01-13 20:54:15--  https://oceandata.sci.gsfc.nasa.gov/ocssw/bundles.sha256sum
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2578 (2.5K) [text/plain]
Saving to: ‘bundles.sha256sum’

bundles.sha256sum   100%[===================>]   2.52K  --.-KB/s    in 0s     

2017-01-13 20:54:16 (295 MB/s) - ‘bundles.sha256sum’ saved [2578/2578]

Installing common (2 of 16)
Installing OCSSW_bash.env (3 of 16)
--2017-01-13 20:54:20--  https://oceandata.sci.gsfc.nasa.gov/ocssw/OCSSW_bash.env
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2894 (2.8K) [text/plain]
Saving to: ‘OCSSW_bash.env’

OCSSW_bash.env      100%[===================>]   2.83K  --.-KB/s    in 0s     

2017-01-13 20:54:21 (362 MB/s) - ‘OCSSW_bash.env’ saved [2894/2894]

Installing README (4 of 16)
--2017-01-13 20:54:21--  https://oceandata.sci.gsfc.nasa.gov/ocssw/README
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3358 (3.3K) [text/plain]
Saving to: ‘README’

README              100%[===================>]   3.28K  --.-KB/s    in 0s     

2017-01-13 20:54:22 (418 MB/s) - ‘README’ saved [3358/3358]

Installing ocrvc (5 of 16)
Installing meris (6 of 16)
Installing modis (7 of 16)
Installing modisa (8 of 16)
Installing hmodisa (9 of 16)
Installing seawifs (10 of 16)
Installing bin (11 of 16)
Installing bin3 (12 of 16)
Installing scripts (13 of 16)
Python version acceptable
Installing seawifs-luts (14 of 16)
Installing aqua-luts (15 of 16)
Error downloading MYD02_Emissive_LUTs.V6.1.35.26_OC2.hdf
Could not communicate with the server.
Please check your network connections and try again later.
Error - Could not install luts for aqua

Its getting stuck in the end . I will be really grateful if anyone of you can help with this issue.

Thanking You,
Swathy.

[gnwiii]

Many open source software providers have moved away from tarballs in favour of direct access to version control repos.  This has important advantages: a) less work for software providers (so they can focus on improving the software), b) users can pull particular versions, and c) lower network costs.  We already have the git bundles, so git can be used to apply recent changes without the need to download a full snapshop of the system.  In our lab I'm hoping we can NFS mount the ocssw directories on a newer (e.g., "Mozilla Modern") system and run install_ocssw.py that way.  I tried this once and all went well until
$ install_ocssw.py -v -b v7.3 --install-dir=/net/<Mint 17 box>/home/<seadas_user>/ocssw.nfs --src --curl --aqua --viirsn
[...]
Checking that an installed executable can run
Checking Python version
Python version acceptable
Installing aqua-luts (14 of 16)
Error downloading MYD02_Emissive_LUTs.V6.1.35.26_OC2.hdf
Could not communicate with the server.
Please check your network connections and try again later.
Error - Could not install luts for aqua


Setting OCSSWROOT to  the NFS directory and manually running update_luts.py worked, but when I go back to the Mint 17 box, getanc.py failed with Error retrieving ancillary file list.  A few minutes later, getanc.py worked.  Still on the Mint 17 box, seadas_benchmark.bash failed with:
===============================================================
Creating L1B file (modis_L1B.py)...
===============================================================

modis_L1B.py A2006167181000.L1A A2006167181000.GEO -o A2006167181000.L1B \
                     --del-hkm --del-qkm
Traceback (most recent call last):
  File "/home/seadas/ocssw.nfs/run/scripts/modis_L1B.py", line 144, in <module>
    sys.exit(main())
  File "/home/seadas/ocssw.nfs/run/scripts/modis_L1B.py", line 137, in main
    modis_env(l1b_instance)
  File "/home/seadas/ocssw.nfs/run/scripts/modules/modis_utils.py", line 398, in modis_env
    '.'.join(self.refl_lut.split('.')[1:5]))
AttributeError: ModisL1B instance has no attribute 'refl_lut'
Command exited with non-zero status 1
so I ran getanc.py -v aqua and the benchmark script worked.

[gnwiii]

@Swathy -- I just had the same problem so it may have been an internet hiccup.  You can try running update_luts.py manually.   I don't think the install_ocssw.py script does anything more after "Python version acceptable" than run update_luts.py for each requested sensor.

[woodbri]

I'm fine with using git over tarballs when I can access git. I have a github account and support multiple projects and have no problem accessing them via Ubuntu 14.04. The problem is that people can not access the oceancolor servers from Ubuntu 14.04 which has an EOL in 2019. And from this thread others seem to be having similar issues so tarballs was a proposal to potentially work around the issue. Hopefully Sean and the IT staff can come up with another idea.

[OB.DAAC - SeanBailey]

One of our sysadmins took on the challenge.  Below is what he found to work...be forewarned it is not an ideal solution, but it is a workable one.

In short, Git will need to be build from source against the openssl version of libcurl instead of the default gnutls.  The following instructions work on Ubuntu
14.04 and should also work for any other OS (although may need modification for different package management systems - e.g. yum).

$ sudo apt-get install build-essential libcurl4-openssl-dev
$ cd ~
$ wget https://github.com/git/git/archive/v2.11.0.zip -O git_2.11.0.zip
$ unzip git_2.11.0.zip
$ cd git-2.11.0
$ make prefix=~/usr/local all && make prefix=~/usr/local install

NOTE: this locally built version of GIT is installed in ~/usr/local/bin/ and your shell environment PATH should be modified to point to
the local version of Git. For example, export the PATH variable in bash:
$ export PATH=~/usr/local/bin:$PATH

$ which git
/home/$USERNAME/usr/local/bin/git

$ git --version
git version 2.11.0

$ git ls-remote
"https://oceandata.sci.gsfc.nasa.gov/git/obpgpyutils/" HEAD
be40eb6871cf353024c22bdd8e4b014b91bc414a        HEAD

As you can see, Git is able to connect to OBPG NASA and negotiate the SSL Certificate as well as the TLS Cipher.
The HEAD request shows the current hash of the git repository.

[OB SeaBASS - inymar]

Hi,

I have been trying to updated to the https. I downloaded "install_ocssw.py" and entered the following commands

chmod +x install_ocssw.py
./install_ocssw.py --install-dir=$HOME/ocssw --git-branch=v7.3 --aqua --seawifs
./install_ocssw.py --install-dir=$HOME/ocssw --git-branch=v7.3 --src
./install_ocssw.py --install-dir=$HOME/ocssw --direct-broadcast

I had to add "--no-check-certificate" in the wget line inside "install_ocssw.py" ...After that it seem to work perfectly. But something went wrong because the LUT update still looking for an http site instead of https.

This is the error message when trying to process a L1A file ....

>=====>  generating geolocation file from modis L1A standard resolution bands...
** Files utcpole.dat/leapsec.dat are more than 2w old
** Running update_luts to update files..

[ MODIS ]
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection error, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 301 - Moved Permanently
* ERROR: The download failed with status code: 301
* Please check your network connection and for the existence of the remote file:
* oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/leapsec.dat

I also try to update the luts manually and got this error.      "./update_luts.py aqua"
URL attempted: oceandata.sci.gsfc.nasa.gov
Well, this is embarrassing...an error occurred that we just cannot get past...
Here is what we know: timed out
Please retry this request at a later time.
* ERROR: The download failed with status code: 301
* Please check your network connection and for the existence of the remote file:
* oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/leapsec.dat
^CWell, the server did not like this...reports: Moved Permanently
* ERROR: The download failed with status code: 301
* Please check your network connection and for the existence of the remote file:
* oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/utcpole.dat
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.

Any help will be greatly appreciated!