SeaDAS 8 install_OCSSW problem with python requests

Please enter here to ask a question about any NASA Science related topics!
OB SeaDAS - dshea
Subject Matter Expert
Subject Matter Expert
Posts: 203
Joined: Thu Mar 05, 2009 10:25 am America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by OB SeaDAS - dshea » Mon Mar 22, 2021 8:49 am America/New_York

Since you are running the installer from inside the GUI, can you post the output of the SeaDAS menu item:

SeaDAS-OCSSW -> SeaDAS/System Info...

don

Tags:

daurin
Posts: 21
Joined: Thu Apr 22, 2010 8:06 am America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by daurin » Mon Mar 22, 2021 9:41 am America/New_York

Hi Don,

I'm actually trying to run the installer from command line because I want Sentinel 3-B/3-B OLCI support which is not shown in the GUI as an option for some reason (see original post for command line options). I tried the GUI as a fallback with the same result.

Here's the output from Seadas/System Information:

Main Application Platform:
Application Version: SeaDAS 8.0.0 (Beta)
Installation Directory: /Users/daurin/Applications/seadas8
Data Directory: /Users/daurin/.seadas8
Configuration: /Users/daurin/Applications/seadas8/etc/snap.properties
WARNING!! File '/Users/daurin/Applications/seadas8/etc/snap.properties' does not exist
Configuration2: /Users/daurin/.seadas8/etc/snap.properties
VM Configuration: /Users/daurin/Applications/seadas8/bin/pconvert.vmoptions
WARNING!! File '/Users/daurin/Applications/seadas8/bin/pconvert.vmoptions' does not exist
VM Configuration (gpt): /Users/daurin/Applications/seadas8/bin/gpt.vmoptions
WARNING!! File '/Users/daurin/Applications/seadas8/bin/gpt.vmoptions' does not exist
Desktop Specification Version: 8.0.1
Engine Specification Version: 8.0.1
JRE: Java(TM) SE Runtime Environment 1.8.0_201-b09
JVM: Java HotSpot(TM) 64-Bit Server VM by Oracle Corporation
Memory: 3641 MiB

OCSSWROOT (Java Env): null

Python3 Directory:
/Users/daurin/anaconda3/bin/python3

SeaDAS Toolbox:
SeaDAS Toolbox Specification Version: 1.0.0
Configuration: /Users/daurin/.seadas8/etc/seadas.properties
OCSSW Root Directory: /Users/daurin/Applications/seadas8/ocssw
OCSSW Log Directory: /
OCSSW Location: local
Environment {$OCSSWROOT} (external): null
WARNING!: An environment variable for OCSSWROOT exists which does not match the GUI configuration. The GUI will use '/Users/daurin/Applications/seadas8/ocssw' as the ocssw root inside the GUI.

-----------------------------------------------

NASA Science Processing (OCSSW):
Environment {$OCSSWROOT}: /Users/daurin/Applications/seadas8/ocssw
Environment {$OCDATAROOT}: /Users/daurin/Applications/seadas8/ocssw/share
l2gen version: msl12 9.5.1-V2021.1 (Jan 27 2021 10:11:03)
l2bin version: 6.2.0
l3bin version: 5.13
l3mapgen version: l3mapgen 2.3.0-V2021.1 (Jan 27 2021 10:10:06)
Installed Missions: ['aqua', 'terra', 'seawifs', 'viirsn', 'viirsj1']

General System and Software:
Operating system: MacOS 10.15.7
Python version: 3.8.3
Java version: 1.8.0_201

Here again is entry in my .bashrc:
export OCSSWROOT="/Users/daurin/Applications/seadas8/ocssw"
source $OCSSWROOT/OCSSW_bash.env

gnwiii
Posts: 566
Joined: Fri Jan 29, 2021 5:51 pm America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by gnwiii » Mon Mar 22, 2021 10:01 am America/New_York

[quote=daurin post_id=7761 time=1616191673 user_id=2258]
Okay, thanks. That was what I had done.

(base) daurin@GSSLA18091785:~/Applications/seadas8/ocssw/opt/bin$ which python3
/Users/daurin/anaconda3/bin/python3

Here's the link:
(base) daurin@GSSLA18091785:~/Applications/seadas8/ocssw/opt/bin$ ll python3
lrwxr-xr-x 1 daurin staff 35 Mar 19 16:02 python3@ -> /Users/daurin/anaconda3/bin/python3

No problem importing requests when I run python, python3, ~/anaconda3/bin/python, ~/anaconda3/bin/python3, nor ~/Applications/seadas8/opt/bin/python3, only if I run python2 and install_ocssw is still throwing ModuleNotFoundError when it tries to import requests.
[/quote]
Is "~/Applications/seadas8/opt/bin/python3" a typo"? The above otherwise looks OK, so a bit of a puzzle. Here (on linux) the first line of "install_ocssw" is "#! /usr/bin/env python3". Either "install_ocssw" is somehow not seeing the PATH setting or Anaconda needs some extra settings.

Are you running "install_ocssw" in the Anaconda "(base)" environment?

How are you "sourcing" "~/Applications/seadas8/ocssw/OCSSW_bash.env"? For most people, it is enough to add the 2 lines to the shell startup script, but it can be a bit tricky to get the right name for the script if you aren't on Ubuntu linux.

What happens with "~/anaconda3/bin/python3 ~/Applications/seadas8/opt/bin"? It is possible that Anaconda's base environment sets some variables to tell python where to find libraries.

Is your shell zsh or bash? I think recent macOS default is "zsh" (and the included "bash" is very old). The 2021 "OCSSW_bash.env" works with zsh on linux.

gnwiii
Posts: 566
Joined: Fri Jan 29, 2021 5:51 pm America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by gnwiii » Mon Mar 22, 2021 10:02 am America/New_York

daurin wrote:
> Hi Don,
>
> I'm actually trying to run the installer from command line because I want
> Sentinel 3-B/3-B OLCI support which is not shown in the GUI as an option
> for some reason (see original post for command line options). I tried the
> GUI as a fallback with the same result.
>
> Here's the output from Seadas/System Information:
> [...]
> Main Application Platform:
> Application Version: SeaDAS 8.0.0 (Beta)
> Installation Directory: /Users/daurin/Applications/seadas8
> Data Directory: /Users/daurin/.seadas8
> Configuration: /Users/daurin/Applications/seadas8/etc/snap.properties
> WARNING!! File '/Users/daurin/Applications/seadas8/etc/snap.properties'
> does not exist
> Configuration2: /Users/daurin/.seadas8/etc/snap.properties
> VM Configuration: /Users/daurin/Applications/seadas8/bin/pconvert.vmoptions
> WARNING!! File '/Users/daurin/Applications/seadas8/bin/pconvert.vmoptions'
> does not exist
> VM Configuration (gpt):
> /Users/daurin/Applications/seadas8/bin/gpt.vmoptions
> WARNING!! File '/Users/daurin/Applications/seadas8/bin/gpt.vmoptions' does
> not exist
> Desktop Specification Version: 8.0.1
> Engine Specification Version: 8.0.1
> JRE: Java(TM) SE Runtime Environment 1.8.0_201-b09
> JVM: Java HotSpot(TM) 64-Bit Server VM by Oracle Corporation
> Memory: 3641 MiB
>
> OCSSWROOT (Java Env): null
>
> Python3 Directory:
> /Users/daurin/anaconda3/bin/python3
>
> SeaDAS Toolbox:
> SeaDAS Toolbox Specification Version: 1.0.0
> Configuration: /Users/daurin/.seadas8/etc/seadas.properties
> OCSSW Root Directory: /Users/daurin/Applications/seadas8/ocssw
> OCSSW Log Directory: /
> OCSSW Location: local
> Environment {$OCSSWROOT} (external): null
> WARNING!: An environment variable for OCSSWROOT exists which does not match
> the GUI configuration. The GUI will use
> '/Users/daurin/Applications/seadas8/ocssw' as the ocssw root inside the
> GUI.

This may point to the problem.

>
> -----------------------------------------------
>
> NASA Science Processing (OCSSW):
> Environment {$OCSSWROOT}: /Users/daurin/Applications/seadas8/ocssw
> Environment {$OCDATAROOT}: /Users/daurin/Applications/seadas8/ocssw/share

This looks correct. What do you get in a terminal for "echo $OCSSWROOT"? There could be an "invisible" or "look-alike" character causing the problem. I've seen this when a user edited "~/.bashrc" with Apple's editor. If you know "vi" or emacs that may reveal the problem.

> l2gen version: msl12 9.5.1-V2021.1 (Jan 27 2021 10:11:03)
> l2bin version: 6.2.0
> l3bin version: 5.13
> l3mapgen version: l3mapgen 2.3.0-V2021.1 (Jan 27 2021 10:10:06)
> Installed Missions: ['aqua', 'terra', 'seawifs', 'viirsn', 'viirsj1']
>
> General System and Software:
> Operating system: MacOS 10.15.7
> Python version: 3.8.3
> Java version: 1.8.0_201
>

Some "~/.bashrc" files have different sections for interactive an batch shells,
or your shell could be "zsh".

> Here again is entry in my .bashrc:
> export OCSSWROOT="/Users/daurin/Applications/seadas8/ocssw"
> source $OCSSWROOT/OCSSW_bash.env

The GUI may not see these settings until you log out and back in.
Last edited by gnwiii on Mon Mar 22, 2021 12:34 pm America/New_York, edited 1 time in total.

daurin
Posts: 21
Joined: Thu Apr 22, 2010 8:06 am America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by daurin » Mon Mar 22, 2021 10:48 am America/New_York

Hi George,

No, not a typo. ~/Applications/seadas8/ocssw/opt/bin/python3 is the symbolic link you suggested I place, so I also tested that for it's ability to run the proper version of python (with requests).

The first line of install_ocssw here is also #!/usr/bin/env python3.

I am running install_ocssw from bash. Here are my shell particulars:
(base) daurin@GSSLA18091785:~$ cat /etc/shells
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells.

/bin/bash
/bin/csh
/bin/ksh
/bin/sh
/bin/tcsh
/bin/zsh
/usr/local/bin/bash
(base) daurin@GSSLA18091785:~$ echo "$SHELL ($0)"
/usr/local/bin/bash (-bash)
(base) daurin@GSSLA18091785:~$ bash --version
GNU bash, version 5.0.18(1)-release (x86_64-apple-darwin18.7.0)

I am sourcing ~/Applications/seadas8/ocssw/OCSSW_bash.env as per my previous post:
>Here again is the entry in my .bashrc:
>export OCSSWROOT="/Users/daurin/Applications/seadas8/ocssw"
>source $OCSSWROOT/OCSSW_bash.env

~/anaconda3/bin/python3 runs python3. ~/Applications/seadas8/opt/bin is not a valid path.

I'm not unclear on your question regarding Anaconda's base environment setting python libraries, but I know that any way I can think of to call up python3, I can import requests.

gnwiii
Posts: 566
Joined: Fri Jan 29, 2021 5:51 pm America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by gnwiii » Mon Mar 22, 2021 12:40 pm America/New_York

> ~/anaconda3/bin/python3 runs python3. ~/Applications/seadas8/opt/bin is not a valid path.

Sorry, I should have said: "$OCSSWROOT/opt/bin". On linux, that directory has the 3rd party tools.

daurin
Posts: 21
Joined: Thu Apr 22, 2010 8:06 am America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by daurin » Mon Mar 22, 2021 2:02 pm America/New_York

I see. Yes, that is the path in which I placed the symbolic link to python3 (~/Applications/seadas8/ocssw/opt/bin). It launches python3 with access to requests.

I wonder if the SeaDAS/System Information I posted above (including some warnings and null values) may contain clues to the problem. This bit in particular, maybe?
OCSSW Root Directory: /Users/daurin/Applications/seadas8/ocssw
OCSSW Log Directory: /
OCSSW Location: local
Environment {$OCSSWROOT} (external): null
WARNING!: An environment variable for OCSSWROOT exists which does not match the GUI configuration. The GUI will use '/Users/daurin/Applications/seadas8/ocssw' as the ocssw root inside the GUI.

This is puzzling, because that is exactly what the OCSSWROOT environmental variable is set to.

I think my next step will be to completely uninstall 8 and rebuild it from scratch.

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 34
Joined: Tue Feb 09, 2021 5:42 pm America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by OB SeaDAS - xuanyang02 » Mon Mar 22, 2021 3:26 pm America/New_York

Don't worry about
"Environment {$OCSSWROOT} (external): null
WARNING!: An environment variable for OCSSWROOT exists which does not match the GUI configuration. The GUI will use '/Users/daurin/Applications/seadas8/ocssw' as the ocssw root inside the GUI."

SeaDAS/System Info is trying to figure out external $OCSSWROOT from your environmental variable $OCSSWROOT.

After you update your $OCSSWROOT in your .bashrc, the shell from which you run seadas 8 does not know you have updated $OCSSWROOT, therefore seadas 8 still thinks your $OCSSWROOT (external) is null, thus the warning.

Judging from
NASA Science Processing (OCSSW):
Environment {$OCSSWROOT}: /Users/daurin/Applications/seadas8/ocssw
Environment {$OCDATAROOT}: /Users/daurin/Applications/seadas8/ocssw/share
you should be fine regarding $OCSSWROOT.

Bing

gnwiii
Posts: 566
Joined: Fri Jan 29, 2021 5:51 pm America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by gnwiii » Mon Mar 22, 2021 5:44 pm America/New_York

daurin wrote:
> I see. Yes, that is the path in which I placed the symbolic link to python3
> (~/Applications/seadas8/ocssw/opt/bin). It launches python3 with access to
> requests.
>
> I wonder if the SeaDAS/System Information I posted above (including some
> warnings and null values) may contain clues to the problem. This bit in
> particular, maybe?
> OCSSW Root Directory: /Users/daurin/Applications/seadas8/ocssw
> OCSSW Log Directory: /
> OCSSW Location: local
> Environment {$OCSSWROOT} (external): null
> WARNING!: An environment variable for OCSSWROOT exists which does not match
> the GUI configuration. The GUI will use
> '/Users/daurin/Applications/seadas8/ocssw' as the ocssw root inside the
> GUI.
>
> This is puzzling, because that is exactly what the OCSSWROOT environmental
> variable is set to.

Yes. Have you logged out and back in?

> I think my next step will be to completely uninstall 8 and rebuild it from
> scratch.

This may just get you back to the same situation, and others may have the same problem, so it is much better to understand the problem before taking drastic steps.

When you run an OCSSW program from the GUI, the "$OCSSWROOT/bin/ocssw_runner" shell script is used. For debugging similar problems, I make a backup of the script and add some debugging statements of the form "echo OCSSWROOT=$OCSSWROOT >> ~/ocssw_runner.log". You can also try running "/bin/bash -l" to see if it throws an error such as a construct in "~/.bashrc" that Apple's elderly bash doesn't support.

gnwiii
Posts: 566
Joined: Fri Jan 29, 2021 5:51 pm America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by gnwiii » Tue Mar 23, 2021 10:05 am America/New_York

daurin wrote:
> Okay, thanks. That was what I had done.
> [...]
> python2 and install_ocssw is still throwing ModuleNotFoundError when it tries to
> import requests.

This suggests that install_ocssw is not getting the same environment you had when checking that "import requests" is installed. Are you running installs from the GUI or the command line? If you are using the GUI then the problem might be the old bash version Apple provides and which is used by the the "$OCSSWROOT/bin/ocssw_runner" shell script. There are a number of things you can try:

What happens when you run "update_luts" from the command-line? I'm not sure if the symbolic link is enough or if you need to use the anaconda3 base environment.

* running "update_luts" from the command-line

* running "ocssw_runner update_luts -v aqua" from the command-line

* running "ocssw_runner --ocsswroot $OCSSWROOT update_luts -v aqua" from the command-line

Post Reply