seadas 7.4, update_luts.py

[ana_dogliotti]

Hi,

The notebook is new, Ubuntu has been just installed and it's the first time I install SeaDAS, so there is only one ocssw folder.

Everything looks OK with the paths, but stil get this "master" version instead of v7.4

echo $OCSSWROOT
/home/seadas-7.4/ocssw

cd $OCSSWROOT/run/scripts
On branch master
Your branch is up-to-date with «origin/master».
nothing to commit, working directory clean

Ana

[gnwiii]

Maybe the git branch didn't get updated on the initial install.  The bundles (the -l /home/usr/Download option) are from the "Master" branch, so that is what you would get if git isn't working.

You can try running "install_ocssw.py -v -i /home/usr/seadas-7.4/ocssw -b v7.4 --aqua".   The verbose option "-v" should show messages from git
like "Switching to branch 'v7.4'.

The attachment shows the full terminal output.
attachment 1

[ana_dogliotti]

Hi,

It seems there is a problem connecting to a page and downloading... I tried reaching the mentioned https, but an error page appears...

Loading checksum file.
Installing common (2 of 13)
Saved local changes with "git stash"
Updating (fetch) existing repository -  /home/seadas-7.4/ocssw/run/data/common
fatal: unable to access 'https://oceandata.sci.gsfc.nasa.gov/ocssw/common.git/': Empty reply from server
Error - Could not run "cd /home/seadas-7.4/ocssw/run/data/common; git fetch"

Ana

[gnwiii]

Well, at least you are getting some indication of the problem.  Try again -- maybe a NASA server or the internet was broken.  

If the retry fails, check that you are using the standard Ubuntu git:


$ which git
/usr/bin/git
$ git --version
git version 2.7.4


[ana_dogliotti]

Indeed, it seems it was a bad moment... now is installing the correct version (git version is OK), but it stopped  somewhere else..
Is it the connection again?
Thanks

Ana

Installing bin (9 of 13)
Saved local changes with "git stash"
Updating (fetch) existing repository -  /home/seadas-7.4/ocssw/run/bin/linux_64
error: GnuTLS recv error (-54): Error in the pull function. (curl_result = 56, http_code = 200, sha1 = 41715799ad8dfd92c251c475f601575c4f38f2b8)
error: Unable to find 41715799ad8dfd92c251c475f601575c4f38f2b8 under https://oceandata.sci.gsfc.nasa.gov/ocssw/bin-linux_64.git
Cannot obtain needed blob 41715799ad8dfd92c251c475f601575c4f38f2b8
while processing commit a34c83362d20420f5233ebb5af38239004c46056.
error: fetch failed.
Error - Could not run "cd /home/seadas-7.4/ocssw/run/bin/linux_64; git fetch"

[gnwiii]

The error indicates the expected data was not received, but the problem could be in your system, at NASA, or somewhere in between.  Keep trying, and start a detailed log of the problems.  

The first step in network troubleshooting is to pinpoint the location of the problem.  Can you rule out a problem with your PC by trying the same command on another PC using the same network connection?  If you can boot to another OS, do you get the problem with the other OS. 

It may also be helpful to do a small test of git independently of the install_ocssw.py script.

If the network problems continue, you will need to get help from you local IT support group.  The IT group where I work gives priority to problems affecting Windows, so I have made an example using git on Windows, based on the information at Ocean Data Science Software Repositories.  The source code bundle is relatively small, and is handy to have when pondering "How does some_ocssw_program do ...?" questions.  If you don't need the build files, you just delete the directory when you are finished testing git.
attachment 1

[OB.DAAC - SeanBailey]

Ana,
Do not be discouraged but these failures.  While we have tried to make the process easy, network issues can be confounding.   Retry.  Most cases the process will pick up where it left off.  A new install of Ubuntu 16.04 should work, as it is the OS we develop on ...and it works for us :grin:
Sean

[ana_dogliotti]

Thank you all for the encouragement! I finally made it and install SeaDAS.. and it's working!

I'm just wondering why the local installation did not work this time (I've done this before). It's really a good solution to avoid connection and server problems, i.e. to insatall everything locally and then only update the luts. I'd like to suggest to solve this "master" problem when installing things locally so seadas can be installed more easily.
Thanks a lot for your big help!
Best

Ana

[gnwiii]

Patience and persistence are virtues here.   For many of us, it can take multiple attempts over several days to get through an OCSSW install.  Common mistakes after a first attempt fails include running the installer using "sudo" and installing new versions of utilities like git or python.   Check your current versions against the system requirements list or ask on this forum if your version is working for others. 

If you have consistent failures and have verified that your configuration meets the requirements and works for others, you need to involve your local IT support group.     IT will be quicker to respond if if they suspect  a problem in their network rather than with a NASA server.  You can ask a colleague (preferably one with the same system configuration you are using) in a different location to run a test at the same time you try the test on your own system.  IT will  be in a better position to help if you can provide a minimal example using git, python, wget, or curl.  You should also mention Earthdata Login Overview and Earthdata Knowledge Base

A previously posted example for wget:
$ cd /tmp
$ wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies  --auth-no-challenge=on --keep-session-cookies  https://tads1.gesdisc.eosdis.nasa.gov/data/aqua_dprep/PM1ATTNR.003/2016/347/PM1ATTNR.P2016347.0000.003.2016347150040
--2017-09-19 09:19:45--  https://tads1.gesdisc.eosdis.nasa.gov/data/aqua_dprep/PM1ATTNR.003/2016/347/PM1ATTNR.P2016347.0000.003.2016347150040
Loaded CA certificate '/opt/local/share/curl/curl-ca-bundle.crt'
Resolving tads1.gesdisc.eosdis.nasa.gov... xx.xxx.xx.xx, 2001:4d0:241a:4041::59
Connecting to tads1.gesdisc.eosdis.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 462848 (452K) [text/plain]
Saving to: ‘PM1ATTNR.P2016347.0000.003.2016347150040’

PM1ATTNR.P2016347.000 100%[========================>] 452.00K  1.34MB/s    in 0.3s   

2017-09-19 09:19:46 (1.34 MB/s) - ‘PM1ATTNR.P2016347.0000.003.2016347150040’ saved [462848/462848]

The same example using curl:
$ cd/tmp
$ curl -L -n --cookie-jar ~/.urs_cookies --output curl-test https://tads1.gesdisc.eosdis.nasa.gov/data/aqua_dprep/PM1ATTNR.003/2016/347/PM1ATTNR.P2016347.0000.003.2016347150040
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   547  100   547    0     0    547      0  0:00:01 --:--:--  0:00:01  3236
100   353    0   353    0     0    353      0 --:--:-- --:--:-- --:--:--   353
100   300  100   300    0     0    300      0  0:00:01  0:00:01 --:--:--  292k
100  452k  100  452k    0     0   452k      0  0:00:01  0:00:01 --:--:--  452k

Note that if your wget fails but curl works, you can try install_ocssw.py --curl.

To check that git is working:

$ cd /tmp
$ git clone -v -b v7.4 https://oceandata.sci.gsfc.nasa.gov/ocssw/ocrvc.git
Cloning into 'ocrvc'...
$ cd ocrvc/
$ git status
On branch v7.4
Your branch is up-to-date with 'origin/v7.4'.

nothing to commit, working tree clean

Finally, Python2.7 (in this case, macports' version on El Capitan) and httplib.  The ">>>" is the Python input prompt.  Just enter the text after each prompt and press "Enter" to get to the next prompt:

$ python2.7
Python 2.7.13 (default, Apr 27 2017, 09:29:25)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.



>>> import httplib
>>> conn = httplib.HTTPSConnection("oceandata.sci.gsfc.nasa.gov")
>>> conn.request("GET", "/ocssw/index.html")
>>> r1 = conn.getresponse()
>>> print r1.status, r1.reason

200 OK

[gnwiii]

The internet is not what it used to be.  Local installation does avoid some of the connection glitches, but the bundles are the "master" branch so the installer has to use git to update to v7.4.  It would be a big job for the NASA crew to regenerate new bundles for each branch, so I think you will agree their time is better spent improving SeaDAS.  

It may be helpful to mention the problems to your IT people.  They may be able to advice you on the best times to do installations and data downloads. Using command-line tools gives you the opportunity to schedule heavy network tasks for off-peak times using the "at" command.