Page 1 of 1

disabling utcpole.dat/leapsec.dat update

Posted: Mon Jan 30, 2017 6:09 am America/New_York
by cedric
Hey,

As many of us, I can't update the utcpole.dat/leapsec.dat files. I'm doing a pratical exercise with my students using SeaDAS. I don't really care to have the new files. I tried to process the L1A to GEO with clicking on "disable-download" button but it's still try to update the utcpole.dat/leapsec.dat files. How can I really fordid SeaDAS to update anything?

Thanks for the help.

execution exception: java.io.IOException: modis_GEO.py failed with exit code 1.
Check log for more details.
** Files utcpole.dat/leapsec.dat are more than 2w old
** Running update_luts to update files...
[ MODIS ]
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 301 - Moved Permanently
* ERROR: The download failed with status code: 301
* Please check your network connection and for the existence of the remote file:
* oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/leapsec.dat
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 301 - Moved Permanently
* ERROR: The download failed with status code: 301
* Please check your network connection and for the existence of the remote file:
* oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/utcpole.dat
[ Sensor: AQUA ]
Connection interrupted, retrying up to 5 more time(s)
Connection error, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 301 - Moved Permanently
Error downloading oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/
Error: /home/cedric/seadas-7.3.2/ocssw/run/var/modisa/cal/OPER/index.html does not exist

disabling utcpole.dat/leapsec.dat update

Posted: Mon Jan 30, 2017 1:29 pm America/New_York
by bruce
I don't know for sure this would work, but you could try using the linux "touch" command on the files which updates some of the times on a file.  It all depends on how the code checks to see if the files are "old".

disabling utcpole.dat/leapsec.dat update

Posted: Mon Jan 30, 2017 4:01 pm America/New_York
by OB.DAAC - SeanBailey
Cedric,

You need to update your processing code.  The change to HTTPS-only for our data servers is affecting you...

Sean

disabling utcpole.dat/leapsec.dat update

Posted: Tue Jan 31, 2017 5:58 am America/New_York
by cedric
I'm trying to avoid that.

I followed the instructions to update the ocssw files. I have the following errors:
cedric@cedric-VirtualBox:~$ sudo ./install_ocssw.py -v --install-dir=/home/jamet/seadas-7.3.2/ocssw --git-branch=v7.3 --aqua --seawifs

gitBase     = https://oceandata.sci.gsfc.nasa.gov/ocssw/
gitBranch   = v7.3
install dir = /home/jamet/seadas-7.3.2/ocssw
arch        = linux_64

git user.name = "Default Seadas User"
git user.email = "seadas-user@localhost"
Installing bundles.sha256sum (1 of 15)
Downloading bundles.sha256sum
--2017-01-31 11:54:42--  https://oceandata.sci.gsfc.nasa.gov/ocssw/bundles.sha256sum
Résolution de oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)… xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connexion à oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 2578 (2,5K) [text/plain]
Enregistre : «bundles.sha256sum»

bundles.sha256sum         100%[==================================>]   2,52K  --.-KB/s    in 0s     

2017-01-31 11:54:43 (609 MB/s) - «bundles.sha256sum» enregistré [2578/2578]

Loading checksum file.
Installing common (2 of 15)
Saved local changes with "git stash"
Updating (fetch) existing repository -  /home/jamet/seadas-7.3.2/ocssw/run/data/common
Switching to branch -  v7.3
La branche v7.3 est paramétrée pour suivre la branche distante v7.3 depuis origin.
Remise à zéro de la branche 'v7.3'
Votre branche est à jour avec 'origin/v7.3'.
Pulling from remote repository
Already up-to-date.
Installing OCSSW_bash.env (3 of 15)
Downloading OCSSW_bash.env
--2017-01-31 11:54:44--  https://oceandata.sci.gsfc.nasa.gov/ocssw/OCSSW_bash.env
Résolution de oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)… xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connexion à oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 2894 (2,8K) [text/plain]
Enregistre : «OCSSW_bash.env»

OCSSW_bash.env            100%[==================================>]   2,83K  --.-KB/s    in 0s     

2017-01-31 11:54:44 (769 MB/s) - «OCSSW_bash.env» enregistré [2894/2894]

Installing README (4 of 15)
Downloading README
--2017-01-31 11:54:44--  https://oceandata.sci.gsfc.nasa.gov/ocssw/README
Résolution de oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)… xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connexion à oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 3358 (3,3K) [text/plain]
Enregistre : «README»

README                    100%[==================================>]   3,28K  --.-KB/s    in 0s     

2017-01-31 11:54:45 (678 MB/s) - «README» enregistré [3358/3358]

Installing ocrvc (5 of 15)
Saved local changes with "git stash"
Updating (fetch) existing repository -  /home/jamet/seadas-7.3.2/ocssw/run/data/ocrvc
Switching to branch -  v7.3
La branche v7.3 est paramétrée pour suivre la branche distante v7.3 depuis origin.
Remise à zéro de la branche 'v7.3'
Votre branche est à jour avec 'origin/v7.3'.
Pulling from remote repository
Already up-to-date.
Installing modis (6 of 15)
Saved local changes with "git stash"
error: could not lock config file .git/config: Le fichier existe
fatal: Could not set 'remote.origin.url' to 'https://oceandata.sci.gsfc.nasa.gov/ocssw/modis.git'
Updating (fetch) existing repository -  /home/jamet/seadas-7.3.2/ocssw/run/data/modis
fatal: '/home/jamet/seadas-7.3.2/ocssw/modis.bundle' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Error - Could not run "cd /home/jamet/seadas-7.3.2/ocssw/run/data/modis; git fetch"

I'm using ubuntu 16 on virtualbox windows 7. My ubuntu crashed while downloading the modis files. Since that, I have this message error. I uninstalled seadas and re-install it. I turned off the virtualbox. But no success.

I have to confess that it's very frustrating.

Thanks for the help

disabling utcpole.dat/leapsec.dat update

Posted: Tue Jan 31, 2017 6:58 am America/New_York
by OB.DAAC - SeanBailey
The crash seems to have left things messy:

> error: could not lock config file .git/config: Le fichier existe
> fatal: Could not set 'remote.origin.url' to 'https://oceandata.sci.gsfc.nasa.gov/ocssw/modis.git'
> Updating (fetch) existing repository -  /home/jamet/seadas-7.3.2/ocssw/run/data/modis
> fatal: '/home/jamet/seadas-7.3.2/ocssw/modis.bundle' does not appear to be a git repository
> fatal: Could not read from remote repository.


The inability to lock the config file is a problem that may be due to wonky permissions.
The inability to read from the remote repository (in this case the "remote" is the downloaded
modis.bundle file) suggests the file was mangled somehow.   You said the virtual machine
crashed during the download, this may have impacted things.  

Make sure the location you are writing to has sufficient space,  and the user
writing it has write privileges.  Since the VM crashed, you may want to identify the reason
for that - did it run out of resources (e.g. fill up the VM disk?)

Sean

disabling utcpole.dat/leapsec.dat update

Posted: Tue Jan 31, 2017 8:10 am America/New_York
by gnwiii
It is much safer to run install_ocssw.py as a normal user, e.g., without using sudo.  Some linux distros will prevent regular users from completely filling the disk, so when you run out of space you could see a useful error message instead of a crash.   The simplest way to recover is to complete remove your existing (owned by root) occsw directory, check that you have ample free space on your (virtual) disk (e.g., room for 15GB and still have 15% free space on your disk), and install again.  If you are adventurous and willing to run command-line tools, it shouldn't hurt to try changing the ownership back to a regular user (sudo chown -R jamet /home/jamet/seadas-7.3.2/ocssw), removing any .git/config.lock files (find these with find $OCSSWROOT -name config.lock), and running the installer again.

disabling utcpole.dat/leapsec.dat update

Posted: Tue Jan 31, 2017 11:52 am America/New_York
by cedric
Thanks for the help the three of you.

I was able to download all the ocssw files but it's still not working. I'm giving up for today. The export bash command is not working.

Thanks

disabling utcpole.dat/leapsec.dat update

Posted: Wed Feb 01, 2017 7:49 am America/New_York
by comachi
Hi, Sean

I'm having problems for generating GEO file too. In my case I updated manually ocssw and placed the updated utcpole.dat/leapsec.dat files in the right directory manually because it seems the update_luts.py didn't. Then I am stuck in the ancillary data file list as the following result generated by SeaDAS 7.3.2:

execution exception: java.io.IOException: modis_GEO.py failed with exit code 45.
Check log for more details.
Determining required attitude and ephemeris files...
Searching database: /home/seadas/seadas/seadas/ocssw/run/var/ancillary_data.db

Input file: /media/seadas/HD3T/amazon/L0/A2017026164000.L1A_LAC
Start time: 2017026164000
End time: 2017026164459

Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
URL attempted: oceancolor.gsfc.nasa.gov
Well, this is embarrassing...an error occurred that we just cannot get past...
Here is what we know: timed out
Please retry this request at a later time.
Error retrieving ancillary file list
Cannot create geolocation from /media/seadas/HD3T/amazon/L0/A2017026164000.L1A_LAC; exiting.

What can I do to get through this problem?
Many thanks.
Claudia

disabling utcpole.dat/leapsec.dat update

Posted: Thu Feb 02, 2017 4:02 pm America/New_York
by treble
To disable the check you can comment out the following line in modis_GEO.py:
    m.utcleap()
That one checks if leapsec and utcpole are up to date (see modis_GEO_utils.py)