problems running MODIS and S3 processing

[lzoffoli]

Hello!
I just installed seadas v7.5.3 in my computer running with ubuntu 16. I managed to install the ocssw processors manually, however with a connection problem to install luts:

Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json: 400  Bad Request
Error - Could not install luts for aqua

I'm facing two problems now:
1) it seems that the connection problem persists, as if I want to update processors I got this error:
ocssw installation script failed to download.
Please check network connection or 'seadas.ocssw.root' variable in the "seadas.config" file.
possible cause of error: connection timed out (connection timed out)

and when I try to run modis_geo I get something like:
Downloading files into /home/laura/seadas-7.5.3/ocssw/var/modis
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request
...no new files.

I cannot see where the connection problem is, as it could download all the files required during installation via install_ocssw.py. For me it means that there are permissions to access to the network or should I change any variable inside the software?

2) when I try to process OLCI/S3 images via command line, I get this error:
Opening: /home/laura/Desktop/Glenan/S3
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c line 586: Could not create NCDF4 file, /home/laura/Desktop/Glenan/S3 .
-E- l2gen: Error opening /home/laura/Desktop/Glenan/S3 for writing.

But I'm pretty sure that there is permissions for writing into the /home/laura/Desktop/Glenan/S3 directory and I have enough space available (more than 400 Gb)

Any help is very welcome!
thanks a lot in advance,
Laura

[gnwiii]

Some additional details are needed, as Ubuntu 16.04 should (and does for me) work.

Did you install the OCSSW software with your own login or using elevated privileges (e.g., "sudo")?   Ubuntu 14.04 needed some extra tweaks to handle the switch to https.   It might be worth checking to see which library is being used, see Wget, Git or Curl built against older versions of GnuTLS can not connect to HTTPS (topic 6492).   Is your Ubuntu the 16.04 LTS version?  Was it updated from an older version or a fresh install?  Has it been it updated to use the most recent libraries from Ubuntu?  Are you using the Ubuntu supplied python 2.7 or some 3rd party (e.g., anaconda) python?   Do you have a direct internet connection or do you use a proxy?

The OCSSW software uses several different programs (git, Python, curl, wget) to download files, so it is not unusual to see some parts work while other fail.

Please post the full command line you along with any par file used for the OLCI/S3 processing.

[lzoffoli]

Hi gnwiii,
thanks for your reply. First to all, I just realized I'm using Ubuntu 18. I guess in any moment it upgraded and I didn't remember... sorry about that! So, do you think it can be a problem running seadas in Ubuntu 18?
Here there are the details about my system:

Ubuntu 18.04.2 LTS  (it was updated from an older version)
GNU Wget 1.19.4 built on linux-gnu.
git version 2.17.1
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2k zlib/1.2.8
Python 3.6.0 :: Anaconda 4.3.1 (64-bit)

I use a proxy to connect to internet.

and finally, this is the complete line I'm using to process OLCI/S3:
l2gen ifile=/home/laura/Desktop/Glenan/S3/S3A_OL_1_EFR____20160706T103902_20160706T104202_20180222T011653_0179_006_108_2160_LR2_R_NT_002.SEN3/xfdumanifest.xml ofile=/home/laura/Desktop/Glenan/S3/

once again, thanks a lot for your help!
Laura

[gnwiii]

I have an Ubuntu 18.04 system but using Ubuntu's python and no proxy.  The OBPG programs (l2gen, binning, and mapping ) seem to work, but the lower-level MODIS software provided by other teams has problems.

The proxy may be the reason update_luts.py is failing.  There have been some posts in the forum that may help (you can find them by searching for "update_luts.py proxy".

The SeaDAS Requirements document says Python3 support is not yet ready.  You may need to edit the Python scripts in the $OCSSWROOT/scripts directory to change the first line of the python scripts from:
#! /usr/bin/env python

to read:
#! /usr/bin/python2


The other problem is with the command line.   Your ofile value is a directory. You need to append a name for the output file.

I'm happy to help -- as an "absent minded scientist" I use the forums to record details that I may need in the future.  For each person who makes the effort to post details of a problem, there are 10 others who had the same problem but gave up without finding the solution, so you get bonus points for providing three issues in one post.

[lzoffoli]

Thanks gnwiii! that's very kind of you!
So, I changed the first line in all the .py scripts as you suggested. I also added a filename when I'm running l2gen:

l2gen ifile=/home/laura/Desktop/Glenan/S3/S3A_OL_1_EFR____20160706T103902_20160706T104202_20180222T011653_0179_006_108_2160_LR2_R_NT_002.SEN3/xfdumanifest.xml ofile=/home/laura/Desktop/Glenan/S3/20160706/20160706.nc

the processing ran well for OLCI/S3 and it's running for MSIS2A too! n:grin:

Now, let's see the problem with MODIS processing, which still persists. If I run modis_GEO.py  through command line, I retrieve this error:
modis_GEO.py /home/laura/Downloads/A2017197131500.L1A_LAC
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json: 400  Bad Request
MODIS GEO version 6.1.0, built Nov 17 2018 15:44:32
Percent valid data (0.00) is less than threshold (95.00)
ERROR: MODIS geolocation processing failed.

and through the GUI I get:
execution exception: java.io.IOException: modis_GEO.py failed with exit code 1.
Check log for more details.
MODIS GEO version 6.1.0, built Nov 17 2018 15:44:32
** Files utcpole.dat/leapsec.dat are not present on hard disk.
** Running update_luts.py to download the missing files...

Downloading files into /home/laura/seadas-7.5.3/ocssw/var/modis
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request
...no new files.

Downloading files into /home/laura/seadas-7.5.3/ocssw/var/modisa
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json: 400  Bad Request
...no new files.

Downloading files into /home/laura/seadas-7.5.3/ocssw/var/modisa/cal/OPER
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json: 400  Bad Request
...no new files.

Downloading files into /home/laura/seadas-7.5.3/ocssw/var/modisa/xcal/OPER
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json: 400  Bad Request
...no new files.
Determining required attitude and ephemeris files...
Searching database: /home/laura/seadas-7.5.3/ocssw/var/ancillary_data.db
()
Input file: /home/laura/Downloads/A2017197131500.L1A_LAC
Sensor    : modisa
Start time: 2017197131500
End time  : 2017197131959
()
  Found: /home/laura/seadas-7.5.3/ocssw/var/anc/2017/197/PM1EPHND.P2017197.1200.003
  Found: /home/laura/seadas-7.5.3/ocssw/var/anc/2017/197/PM1ATTNR.P2017197.1200.003

Creating MODIS geolocation file...
geogen_modis returned with exit status: 1
Percent valid data (0.00) is less than threshold (95.00)
('Validation test returned with error code: ', SystemExit(1,))
ERROR: MODIS geolocation processing failed.

However, it created folders in ocssw/var/anc and downloaded ATTNR and EPHND files. So, I'm not sure if this problem comes from the version of Ubuntu or a connection problem or both. Something that I saw it was an error to install LUTs (Aqua and SeaWiFS). Maybe this is why I cannot update LUTs and I can't also process MODIS L1A. There is a way to manually install only SeaWIFs and Aqua LUTs?
Anyways, I don't need to process MODIS images right now, but it's always useful to have it working.

Some info about my connection required in some other posts I saw: I'm using a proxy with http.
http_proxy=proxy.sciences.univ-nantes.prive:3128
https_proxy=proxy.sciences.univ-nantes.prive:3128

when I digit git diff > seadas-https-proxy.diff, I get:

Not a git repository
To compare two paths outside a working tree:
usage: git diff [--no-index] <path> <path>

Another question, every time that I want to launch seadas from command line or if I want to run l2gen command, I need to set paths (export PATH=/home/laura/seadas-7.5.3/bin:$PATH and export OCSSWROOT=/home/laura/seadas-7.5.3/ocssw, source $OCSSWROOT/OCSSW_bash.env), otherwise it doesn't recognize the commands. How can I fix the paths?

Sorry for the so long post! and thanks a lot for your help!
Laura

[OB SeaDAS - knowles]

Regarding setting paths for command line usage you can edit your ~/.bashrc file which gets run each time you launch a new terminal.  For instance mine currently contains these lines:

export SEADAS=/seadas/installations/7.5.3
export PATH=$SEADAS/bin:$PATH
export OCSSWROOT=$SEADAS/ocssw
source $OCSSWROOT/OCSSW_bash.env

Regarding the modis_GEO.py failure you either didn't install the Aqua processors or the installation failed? 

From your error message, you are missing these files which come with the Aqua installation:
$OCSSWROOT/var/modis/utcpole.dat
$OCSSWROOT/var/modis/leapsec.dat

For command line the installation for Aqua is:
install_ocssw.py --aqua

Within the GUI it is menu: OCSSW > Update Data Processors
and then select the  aqua checkbox

Danny

[OB SeaDAS - knowles]

To specifically include the git branch (in this case OCSSW branch v7.5 for SeaDAS version 7.5.x):
install_ocssw.py -b v7.5 --aqua

Danny

[gnwiii]

Danny's suggestion should fix the issue with the terminal configuration.   You are not the only one to report the error:

Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request

The earlier report was also using a proxy.  Unfortunately, this post lacks details of the OS and python version, and doesn't provide a solution. 

Previous problems with proxies were due to different servers or ports for http and https, and should be fixed now.  update luts.py works on my Ubuntu 18.04 system (no proxy) so there could still be an issue with the proxy, but we should make sure you have the same python version.  On my system:
$ which python
/usr/bin/python
$ python --version
Python 2.7.15rc1


It appears that update luts.py HAS been upgraded for python3, so it might be useful to try running the script with python3. On my
(noproxy) Ubuntu 18.04 system I get:

$ python3 $OCSSWROOT/scripts/update_luts.py -v  aqua

Downloading files into /home/seadas/ocssw/var/modis
...no new files.

Downloading files into /home/seadas/ocssw/var/modisa
...no new files.

Downloading files into /home/seadas/ocssw/var/modisa/cal/OPER
...no new files.

Downloading files into /home/seadas/ocssw/var/modisa/xcal/OPER
...no new files.

It is possible to download the LUT's on another system and transfer them to the one that is having problems: [https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=5467%5doffline SeaDAS install[/url].

[lzoffoli]

Hi gnwiii and Danny,
Sorry for my late reply. I'm coming back to Seadas now and trying to fix it using your suggestions.

So, I included those lines in the ~./bashrc file:

export SEADASPATH=/home/laura/seadas-7.5.3/bin:$SEADASPATH
export OCSSWROOT=/home/laura/seadas-7.5.3/ocssw
source $OCSSWROOT/OCSSW_bash.env

However, in a new terminal, if I not run export PATH=/home/laura/seadas-7.5.3/bin:$PATH, when I launch seadas it doesn't find the command. Anyways, it's not a big deal

About the installation of ocssw -aqua, I tried with ./install_ocssw.py -b v7.5 --aqua
The installation runs well for 9 of 10 processes. However, it fails in the installation of luts

Installing aqua-luts (10 of 10)
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/cal/OPER/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json: 400  Bad Request
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modisa/xcal/OPER/?format=json: 400  Bad Request
Error - Could not install luts for aqua

Running $ python3 $OCSSWROOT/scripts/update_luts.py -v  aqua I also got:
Bad response for https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/?format=json: 400  Bad Request

So, I manually downloaded Ancillary/LUTs and placed them into .../seadas-7.5.3/ocssw/var as suggested by gnwiii... MODIS/Aqua processing is running beautifully now!

thanks a lot for your help! I'll try with other sensors in this week and I post something if I have any error

cheers!
Laura

[gnwiii]

Normal user installs don't use SEADASPATH.   If you want to run the SeaDAS GUI from a terminal, you can add the directory to the system path in ~/.profile by adding the line:

PATH="/home/laura/seadas-7.5.3/bin:$PATH"

Many users add the SeaDAS GUI to their menus, but the procedure for doing that varies widely across linux versions.  Another approach is to create a
$HOME/bin directory and put a symbolic link to the SeaDAS GUI in this directory:

cd ~/bin
ln -s ../seadas-7.5.3/bin/seadas .


After the next login you should be able to start the seadas GUI from a terminal. 

You seem to have a fairly common proxy configuration, and wget (the default downloader used by install_ocssw.py) works.  It isn't unusual to have the installer work fine until the update LUT's stage, which involves python scripts that download data directly.  These scripts are not as robust as the wget and curl programs, so I'm glad you were able to manually install the LUT's.