Welcome to the Earthdata Forum! Here, the scientific user community and subject matter experts from NASA Distributed Active Archive Centers (DAACs), and other contributors, discuss research needs, data, and data applications.
by 7yl4r » Mon Apr 15, 2019 4:54 pm America/New_York
I run `install_ocssw.py` daily on a few machines to keep things up-to-date and have been logging errors like the following error on all nodes since 2019-03-27:
``` [root@userproc test]# ls /srv/imars-objects/west_fl_pen ls: cannot access /srv/imars-objects/west_fl_pen: No such file or directory [root@userproc test]# /opt/ocssw/install_ocssw.py -v -i /opt/ocssw -g https://oceandata.sci.gsfc.nasa.gov/ocssw/ -a linux_64 --curl --aqua --terra --olci
git user.name = "Default Seadas User" git user.email = "seadas-user@localhost" Installing bundles.sha256sum (1 of 13) Downloading bundles.sha256sum % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3880 100 3880 0 0 12846 0 --:--:-- --:--:-- --:--:-- 12890 Loading checksum file. Installing common (2 of 13) Updating (fetch) existing repository - /opt/ocssw/share/common Switching to branch - v7.5 Branch 'v7.5' set up to track remote branch 'v7.5' from 'origin'. Reset branch 'v7.5' Your branch is up to date with 'origin/v7.5'. Pulling from remote repository Already up to date. Installing OCSSW_bash.env (3 of 13) Downloading OCSSW_bash.env.v7.5 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3001 100 3001 0 0 9985 0 --:--:-- --:--:-- --:--:-- 10003 Installing ocrvc (4 of 13) Updating (fetch) existing repository - /opt/ocssw/share/ocrvc Switching to branch - v7.5 Branch 'v7.5' set up to track remote branch 'v7.5' from 'origin'. Reset branch 'v7.5' Your branch is up to date with 'origin/v7.5'. Pulling from remote repository Already up to date. Installing modis (5 of 13) Updating (fetch) existing repository - /opt/ocssw/share/modis Switching to branch - v7.5 Branch 'v7.5' set up to track remote branch 'v7.5' from 'origin'. Reset branch 'v7.5' Your branch is up to date with 'origin/v7.5'. Pulling from remote repository Already up to date. Installing modis/aqua (6 of 13) Updating (fetch) existing repository - /opt/ocssw/share/modis/aqua Switching to branch - v7.5 Branch 'v7.5' set up to track remote branch 'v7.5' from 'origin'. Reset branch 'v7.5' Your branch is up to date with 'origin/v7.5'. Pulling from remote repository Already up to date. Installing modis/terra (7 of 13) Updating (fetch) existing repository - /opt/ocssw/share/modis/terra fatal: unable to access 'https://oceandata.sci.gsfc.nasa.gov/ocssw/modisterra.git/': Operation timed out after 300009 milliseconds with 0 out of 0 bytes received Error - Could not run "cd /opt/ocssw/share/modis/terra; git fetch" ```
Note that running the command in the error seems to work just fine:
``` [root@userproc terra]# cd /opt/ocssw/share/modis/terra; git fetch [root@userproc terra]# git status On branch v7.5 Your branch is up to date with 'origin/v7.5'. nothing to commit, working tree clean ```
First, you really don't need to update daily. Trust me, we aren't changing things that fast :wink: You could run update_luts.py for the aqua and terra sensors monthly to keep the calibration updates current (or if you're really keen, weekly), but the code can be checked on a much lower frequency.
That said, the error looks like you may be tripping a firewall limit. I just ran your command successfully on my laptop from the comfort of my couch, so it's not install_ocssw.py. You mention, "on all nodes". If all those nodes are coming into our servers from the same IP, and all are running the script at the same time you may want to try cycling through one at a time.
by gnwiii » Mon Apr 15, 2019 8:23 pm America/New_York
I often encounter internet "glitches" running install_ocssw.py, so I expect to run the script multiple times. It might be better to run a separate update for each sensor, or you can omit the sensors that previously succeeded in subsequent attempts. If the script fails in the final stages updating LUT's you can run the update scripts rather than going thru the installer. All this is a bit much to automate.
I tried a version of your command adapted for my configuration on Ubuntu 16.04. There were some problems. The first was recovered, the second resulted in an incomplete update: install_ocssw.py -v -i $HOME/ocssw -g https://oceandata.sci.gsfc.nasa.gov/ocssw/ -a linux_64 --curl --aqua --terra --olci [...] Installing modis/terra (7 of 13) Downloading new directory - /home/seadas/ocssw/share/modis/terra Downloading modisterra.bundle % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:05:00 --:--:-- 0 Warning: Transient problem: timeout Will retry in 5 seconds. 5 retries left. 100 288M 100 288M 0 0 53.2M 0 0:00:05 0:00:05 --:--:-- 61.9M comparing checksum for modisterra.bundle Cloning into '/home/seadas/ocssw/share/modis/terra'... Receiving objects: 100% (177/177), 288.39 MiB | 59.85 MiB/s, done. Resolving deltas: 100% (34/34), done. Checking connectivity... done. Checking out files: 100% (174/174), done. Updating (fetch) existing repository - /home/seadas/ocssw/share/modis/terra From https://oceandata.sci.gsfc.nasa.gov/ocssw/modisterra a4f1114..db7095e master -> origin/master a4f1114..db7095e v7.5 -> origin/v7.5 Switching to branch - v7.5 Branch v7.5 set up to track remote branch v7.5 from origin. Switched to a new branch 'v7.5' Pulling from remote repository - /home/seadas/ocssw/share/modis/terra [...] Installing scripts (11 of 13) Updating (fetch) existing repository - /home/seadas/ocssw/scripts From https://oceandata.sci.gsfc.nasa.gov/ocssw/scripts 739aab2..0ec4351 v7.5 -> origin/v7.5 739aab2..0ec4351 master -> origin/master Switching to branch - v7.5 Branch v7.5 set up to track remote branch v7.5 from origin. Reset branch 'v7.5' Your branch is up-to-date with 'origin/v7.5'. Pulling from remote repository fatal: unable to access 'https://oceandata.sci.gsfc.nasa.gov/ocssw/scripts.git/': Operation timed out after 0 milliseconds with 0 out of 0 bytes received Error - Could not run "cd /home/seadas/ocssw/scripts; git pull --progress"
I notice you installed the OCSSW software as "root". General "best practices" minimize the use of elevated privileges, and newer systems may impose limits on what can be done with "root" privileges (e.g., no GUI's) so can produce weird errors. If you wish to install to /opt/ocssw you can create the directory using root privileges and change ownership to the "seadas" user, then run the installation as the "seadas" user.
by 7yl4r » Tue Apr 16, 2019 10:23 am America/New_York
You are definitely right about install as "root" being in general a bad idea, but it was the quickest hack that got things working on these limited-use systems.
I had only minimal issues with `install_ocssw.py` previous to the aforementioned date and the error now seems quite consistent. Your suggestions are good, but perhaps this automated retry logic should be in the script itself? I don't really have time to debug this right now so in the meantime I have temporarily disabled updates. If anyone has suggestions or a fix is deployed let me know and I will flip the switch and test it.
by 7yl4r » Tue Apr 16, 2019 10:26 am America/New_York
Oops, just saw your response about the firewall. I suspect updating too often and being blocked by a firewall is my issue. I will consider deploying a cronjob instead of using puppet when I have time to revisit this.
