SeaDAS 8 install_OCSSW problem with python requests

Please enter here to ask a question about any NASA Science related topics!
daurin
Posts: 27
Joined: Thu Apr 22, 2010 8:06 am America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by daurin » Thu Mar 25, 2021 10:09 am America/New_York

Here is the whole thing again in a fresh shell:

(base) daurin@GSSLA18091785:~$ echo $PATH
/Users/daurin/anaconda3/bin:/Users/daurin/anaconda3/condabin:/Users/daurin/Applications/seadas8/ocssw/opt/bin:/Users/daurin/Applications/seadas8/ocssw/bin:/Applications/MATLAB_R2020a.app/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin
(base) daurin@GSSLA18091785:~$ which python3
/Users/daurin/anaconda3/bin/python3
(base) daurin@GSSLA18091785:~$ which python
/Users/daurin/anaconda3/bin/python
(base) daurin@GSSLA18091785:~$ which python2
/usr/bin/python2
(base) daurin@GSSLA18091785:~$ python3
Python 3.8.3 (default, Jul 2 2020, 11:26:31)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> exit()
(base) daurin@GSSLA18091785:~$ cd ~/Downloads/
(base) daurin@GSSLA18091785:~/Downloads$ ./install_ocssw --install_dir ~/Applications/seadas8/ocssw --tag V2021.1 --seadas --modisa --seawifs --modist --msis2a --msis2b --olcis3a --olcis3b --viirsj1 --viirsn
Installing (1 of 19) - bin_macosx_intel
Installing (2 of 19) - lib_macosx_intel
Installing (3 of 19) - opt_macosx_intel
Installing (4 of 19) - root
Installing (5 of 19) - common
Installing (6 of 19) - modis
Installing (7 of 19) - modisa
Installing (8 of 19) - modist
Installing (9 of 19) - msi
Installing (10 of 19) - msis2a
Installing (11 of 19) - msis2b
Installing (12 of 19) - ocrvc
Installing (13 of 19) - olci
Installing (14 of 19) - olcis3a
Installing (15 of 19) - olcis3b
Installing (16 of 19) - seawifs
Installing (17 of 19) - viirs
Installing (18 of 19) - viirsj1
Installing (19 of 19) - viirsn
Installing lut - modisa
Traceback (most recent call last):
File "/Users/daurin/Applications/seadas8/ocssw/bin/update_luts", line 9, in <module>
import seadasutils.LutUtils as Lut
File "/Users/daurin/Applications/seadas8/ocssw/bin/seadasutils/LutUtils.py", line 3, in <module>
import seadasutils.JsonUtils as Session
File "/Users/daurin/Applications/seadas8/ocssw/bin/seadasutils/JsonUtils.py", line 6, in <module>
from seadasutils.ProcUtils import getSession, httpdl
File "/Users/daurin/Applications/seadas8/ocssw/bin/seadasutils/ProcUtils.py", line 13, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
Error: return = 1 : trying to run command = /Users/daurin/Applications/seadas8/ocssw/bin/ocssw_runner --ocsswroot /Users/daurin/Applications/seadas8/ocssw update_luts modisa

(base) daurin@GSSLA18091785:~/Downloads$ ocssw_runner --ocsswroot $OCSSWROOT env
LIB3_DIR=/Users/daurin/Applications/seadas8/ocssw/opt
PROJ_LIB=/Users/daurin/Applications/seadas8/ocssw/opt/share/proj
LIB3_CHECK=0
TERM_PROGRAM=Apple_Terminal
TERM=xterm-256color
SHELL=/usr/local/bin/bash
LIB3_LIB=/Users/daurin/Applications/seadas8/ocssw/opt/lib
TMPDIR=/var/folders/th/7q7vkd_x68g_88b8j5qg5kxh0000gp/T/
CONDA_SHLVL=1
CONDA_PROMPT_MODIFIER=(base)
TERM_PROGRAM_VERSION=433
TERM_SESSION_ID=3829E37E-4F73-4A24-8052-B9280A852AED
LIB3_BIN=/Users/daurin/Applications/seadas8/ocssw/opt/bin
LIB3_INC=/Users/daurin/Applications/seadas8/ocssw/opt/include
USER=daurin
SWTBL=/Users/daurin/Applications/seadas8/ocssw/share/seawifs
CONDA_EXE=/Users/daurin/anaconda3/bin/conda
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.h63i0khAMj/Listeners
OCVARROOT=/Users/daurin/Applications/seadas8/ocssw/var
OCSSW_ARCH=macosx_intel
_CE_CONDA=
ASAP_PARMS=/Users/daurin/Applications/seadas8/ocssw/share/common/nav
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/daurin/anaconda3/bin:/Users/daurin/anaconda3/condabin:/Users/daurin/Applications/seadas8/ocssw/opt/bin:/Users/daurin/Applications/seadas8/ocssw/bin:/Applications/MATLAB_R2020a.app/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin
L2GEN_ANC=/Users/daurin/Applications/seadas8/ocssw/var/anc
NAVCTL=/Users/daurin/Applications/seadas8/ocssw/share/seawifs/nav
OCSSWROOT=/Users/daurin/Applications/seadas8/ocssw
CONDA_PREFIX=/Users/daurin/anaconda3
HDFEOS_LIB=/Users/daurin/Applications/seadas8/ocssw/opt/EOS/lib/macintel64
OCSSW_BIN=/Users/daurin/Applications/seadas8/ocssw/bin
HRPT_STATION_IDENTIFICATION_FILE=/Users/daurin/Applications/seadas8/ocssw/share/seawifs/station/station_id.dat
PWD=/Users/daurin/Downloads
LANG=en_US.UTF-8
OCSSW_SVN=localhost
XPC_FLAGS=0x0
OCDATAROOT=/Users/daurin/Applications/seadas8/ocssw/share
SWFTBL=/Users/daurin/Applications/seadas8/ocssw/share/seawifs/qc
XPC_SERVICE_NAME=0
OCSSW_DEBUG=0
PGSLIB=/Users/daurin/Applications/seadas8/ocssw/opt/EOS/lib/macintel64
CXX=g++
_CE_M=
SHLVL=2
HOME=/Users/daurin
FC=gfortran
CONDA_PYTHON_EXE=/Users/daurin/anaconda3/bin/python
LOGNAME=daurin
ORBCTL=/Users/daurin/Applications/seadas8/ocssw/share/seawifs/nav
GCC_TUNE=
CONDA_DEFAULT_ENV=base
OCTS_REGISTRATION_TABLES=/Users/daurin/Applications/seadas8/ocssw/share/octs/nav
CC=gcc
DISPLAY=/private/tmp/com.apple.launchd.LsyV81Pece/org.xquartz:0
EOS_LIB_PREFIX=macintel64
ELEMENTS=/Users/daurin/Applications/seadas8/ocssw/var/seawifs
PGSINC=/Users/daurin/Applications/seadas8/ocssw/opt/EOS/include
NAVQC=/Users/daurin/Applications/seadas8/ocssw/share/seawifs/nav
_=/usr/bin/env
(base) daurin@GSSLA18091785:~/Downloads$ echo $PATH
/Users/daurin/anaconda3/bin:/Users/daurin/anaconda3/condabin:/Users/daurin/Applications/seadas8/ocssw/opt/bin:/Users/daurin/Applications/seadas8/ocssw/bin:/Applications/MATLAB_R2020a.app/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin
(base) daurin@GSSLA18091785:~/Downloads$

This thread has exploded, so I'm going to work with Don offline to trace the problem, and we'll post the results here. Thanks to you both.

Dirk

Tags:

daurin
Posts: 27
Joined: Thu Apr 22, 2010 8:06 am America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by daurin » Fri Apr 02, 2021 9:21 am America/New_York

To follow up, Don and I were unable to trace the problem, but we did create a workaround.

With my .bashrc and .bash_profile set up as they should be, update_luts (when called by install_ocssw) was and is calling a version of python that did not have the requests module.

There are three pythons on my computer: macOS' default python2 (usr/bin/python2, v2.7.16), macOS' default python3 (/usr/bin/python3, v3.8.2), and anaconda's python3 (/User/daurin/anaconda3/bin/python3.8 aka python3 aka python.app aka python, v3.8.3). Both versions of python3 have requests, but python2 does not. Neither of us have any idea how update_luts was calling python2, but it appeared to be.

What worked was adding a line in .bashrc to force anaconda3/bin to the front of the path (again) before setting up the OCSSW environment. Anaconda already has a code block in .bash_profile that puts anaconda/bin before /usr/anything, but only by doing it AGAIN in the .bashrc does install_ocssw work properly. So, in the .bashrc, add:

export PATH="/Users/daurin/anaconda3/bin:$PATH"
before the usual SeaDAS stuff:
export OCSSWROOT="/Users/daurin/Applications/seadas8/ocssw"
source $OCSSWROOT/OCSSW_bash.env

Now, my path looks like this (note the redundant listing for /Users/daurin/anaconda3/bin):

/Users/daurin/Applications/seadas8/ocssw/opt/bin:/Users/daurin/Applications/seadas8/ocssw/bin:/Users/daurin/anaconda3/bin:/Applications/MATLAB_R2020a.app/bin:/Users/daurin/anaconda3/bin:/Users/daurin/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin

We cannot explain why this works, but I, for one, am moving on for now.

gnwiii
Posts: 654
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

Re: SeaDAS 8 install_OCSSW problem with python requests

by gnwiii » Fri Apr 02, 2021 3:06 pm America/New_York

I'm glad you got it working. There are, however, other people having problems with python for ESA SNAP snappy on macOS, and I expect SeaDAS 8 users will also get snappy with the final release, so it would be nice to understand how macOS manages the PATH. It seems to me that the default path is somehow being added after "~/.bashrc" is run at startup. I will pursue this in Apple Developers forums.

I recall that you also had a "/usr/local/bin/python" (have you installed python with homebrew?).

daurin
Posts: 27
Joined: Thu Apr 22, 2010 8:06 am America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by daurin » Mon Apr 05, 2021 1:34 pm America/New_York

Yes, you are right! There is yet /another/ python3 on this machine I was forgetting. Origin unknown. The links in /usr/local/bin ("python3" and "python3.9") point to usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/bin/python3.9.

And here is the thing: if I run THAT python3, it does not have the requests module; I get the ModuleNotFoundError that started this thread! I think you're onto something.

bcoll018
Posts: 2
Joined: Tue Nov 09, 2021 1:41 pm America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by bcoll018 » Tue Nov 09, 2021 1:59 pm America/New_York

Has any more progress been made on this issue? I am having this same problem with update_luts using SeaDAS 8.1.0 through the GUI, as well as through the command line prompt (Mac Big Sur 11.16). The workaround posted by daurin does not solve this problem for me. I can import "requests" using python3 and python, but it cannot be found when called by update_luts. Maybe it is calling python2 somehow? Any thoughts?

I included my $PATH below as well as the location of my python versions.

(base) bcollist@LALAL0721030123 scripts % which python
/Users/bcollist/opt/anaconda3/bin/python

(base) bcollist@LALAL0721030123 scripts % which python3
/Users/bcollist/opt/anaconda3/bin/python3

(base) bcollist@LALAL0721030123 scripts % which python2
/usr/bin/python2

(base) bcollist@LALAL0721030123 scripts % echo $PATH
/Users/bcollist/opt/anaconda3/bin:/Users/bcollist/Applications/Seadas/ocssw/opt/bin:/Users/bcollist/Applications/Seadas/ocssw/bin:/Users/bcollist/opt/anaconda3/bin:/Users/bcollist/opt/anaconda3/bin:/Users/bcollist/opt/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 200
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1

Re: SeaDAS 8 install_OCSSW problem with python requests

by OB SeaDAS - xuanyang02 » Tue Nov 09, 2021 2:41 pm America/New_York

In your SeaDAS GUI, can you do SeaDAS-OCSSW -> SeaDAS/System Info and copy and paste the result here?

bcoll018
Posts: 2
Joined: Tue Nov 09, 2021 1:41 pm America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by bcoll018 » Tue Nov 09, 2021 3:14 pm America/New_York

Here you go!

-Brian

Main Application Platform:
Application Version: SeaDAS 8.1.0
Installation Directory: /Users/bcollist/Applications/Seadas
Data Directory: /Users/bcollist/.seadas8
Configuration: /Users/bcollist/Applications/Seadas/etc/snap.properties
WARNING!! File '/Users/bcollist/Applications/Seadas/etc/snap.properties' does not exist
Configuration2: /Users/bcollist/.seadas8/etc/snap.properties
VM Configuration: /Users/bcollist/Applications/Seadas/bin/pconvert.vmoptions
WARNING!! File '/Users/bcollist/Applications/Seadas/bin/pconvert.vmoptions' does not exist
VM Configuration (gpt): /Users/bcollist/Applications/Seadas/bin/gpt.vmoptions
WARNING!! File '/Users/bcollist/Applications/Seadas/bin/gpt.vmoptions' does not exist
Desktop Specification Version: 8.0.3
Engine Specification Version: 8.0.3
JRE: Java(TM) SE Runtime Environment 1.8.0_201-b09
JVM: Java HotSpot(TM) 64-Bit Server VM by Oracle Corporation
Memory: 3641 MiB

OCSSWROOT (Java Env): null

Python3 Directory:
/Users/bcollist/opt/anaconda3/bin/python3

SeaDAS Toolbox:
SeaDAS Toolbox Specification Version: 1.1.0
Configuration: /Users/bcollist/.seadas8/etc/seadas.properties
OCSSW Root Directory: /Users/bcollist/Applications/Seadas/ocssw
OCSSW Log Directory: /
OCSSW Location: local
Environment {$OCSSWROOT} (external): null

-----------------------------------------------

NASA Science Processing (OCSSW):
Environment {$OCSSWROOT}: /Users/bcollist/Applications/Seadas/ocssw
Environment {$OCDATAROOT}: /Users/bcollist/Applications/Seadas/ocssw/share
l2gen version: msl12 9.5.1-V2021.2 (Jun 18 2021 11:21:15)
l2bin version: 7.0.1
l3bin version: 5.13
l3mapgen version: l3mapgen 2.3.0-V2021.2 (Jun 18 2021 11:22:30)
Installed Missions: ['aqua', 'terra', 'seawifs', 'viirsn', 'viirsj1']

General System and Software:
Operating system: MacOS 10.16
Java version: 11.0.11
Python3 version: 3.8.8
Python3 Path: /Users/bcollist/opt/anaconda3/bin/python3
Env: ASAP_PARMS=/Users/bcollist/Applications/Seadas/ocssw/share/common/nav
CC=gcc
COMMAND_MODE=unix2003
CONDA_DEFAULT_ENV=base
CONDA_EXE=/Users/bcollist/opt/anaconda3/bin/conda
CONDA_PREFIX=/Users/bcollist/opt/anaconda3
CONDA_PROMPT_MODIFIER=(base)
CONDA_PYTHON_EXE=/Users/bcollist/opt/anaconda3/bin/python
CONDA_SHLVL=1
CXX=g++
ELEMENTS=/Users/bcollist/Applications/Seadas/ocssw/var/seawifs
EOS_LIB_PREFIX=macintel64
FC=gfortran
GCC_TUNE=
HDFEOS_LIB=/Users/bcollist/Applications/Seadas/ocssw/opt/EOS/lib/macintel64
HOME=/Users/bcollist
HRPT_STATION_IDENTIFICATION_FILE=/Users/bcollist/Applications/Seadas/ocssw/share/seawifs/station/station_id.dat
J2D_PIXMAPS=shared
JAVA_MAIN_CLASS_61535=org.netbeans.Main
L2GEN_ANC=/Users/bcollist/Applications/Seadas/ocssw/var/anc
LC_CTYPE=UTF-8
LIB3_BIN=/Users/bcollist/Applications/Seadas/ocssw/opt/bin
LIB3_CHECK=0
LIB3_DIR=/Users/bcollist/Applications/Seadas/ocssw/opt
LIB3_INC=/Users/bcollist/Applications/Seadas/ocssw/opt/include
LIB3_LIB=/Users/bcollist/Applications/Seadas/ocssw/opt/lib
LOGNAME=bcollist
NAVCTL=/Users/bcollist/Applications/Seadas/ocssw/share/seawifs/nav
NAVQC=/Users/bcollist/Applications/Seadas/ocssw/share/seawifs/nav
OCDATAROOT=/Users/bcollist/Applications/Seadas/ocssw/share
OCSSWROOT=/Users/bcollist/Applications/Seadas/ocssw
OCSSW_ARCH=macosx_intel
OCSSW_BIN=/Users/bcollist/Applications/Seadas/ocssw/bin
OCSSW_DEBUG=0
OCSSW_SVN=localhost
OCTS_REGISTRATION_TABLES=/Users/bcollist/Applications/Seadas/ocssw/share/octs/nav
OCVARROOT=/Users/bcollist/Applications/Seadas/ocssw/var
ORBCTL=/Users/bcollist/Applications/Seadas/ocssw/share/seawifs/nav
PATH=/Users/bcollist/Applications/Seadas/ocssw/opt/bin:/Users/bcollist/Applications/Seadas/ocssw/bin:/Users/bcollist/opt/anaconda3/bin:/Users/bcollist/opt/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
PGSINC=/Users/bcollist/Applications/Seadas/ocssw/opt/EOS/include
PGSLIB=/Users/bcollist/Applications/Seadas/ocssw/opt/EOS/lib/macintel64
PROJ_LIB=/Users/bcollist/Applications/Seadas/ocssw/opt/share/proj
PWD=/
SHELL=/bin/zsh
SHLVL=3
SQLITE_EXEMPT_PATH_FROM_VNODE_GUARDS=/Users/bcollist/Library/WebKit/Databases
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.dD8iHwqy1n/Listeners
SWFTBL=/Users/bcollist/Applications/Seadas/ocssw/share/seawifs/qc
SWTBL=/Users/bcollist/Applications/Seadas/ocssw/share/seawifs
TMPDIR=/var/folders/_s/1_4wbjps7sz5snkvxgrt6mv80000gp/T/
USER=bcollist
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
_=/usr/bin/env
_CE_CONDA=
_CE_M=
__CFBundleIdentifier=gov.nasa.gsfc.seadas
__CF_USER_TEXT_ENCODING=0x1F6:0x0:0x0

gnwiii
Posts: 654
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

Re: SeaDAS 8 install_OCSSW problem with python requests

by gnwiii » Tue Nov 09, 2021 7:00 pm America/New_York

bcoll018 wrote: Tue Nov 09, 2021 1:59 pm America/New_York Has any more progress been made on this issue?
This isn't something that can be fixed in SeaDAS. Apple recommends that users avoid using the "system" python for applications because that usually requires changes to the installed packages. It is up to the user to ensure that the SeaDAS requirements are met. Unfortunately, the internet is full of misleading advice, so you need to understand the SeaDAS and OCSSW requirements and then work with your Python packager's documentation to ensure that SeaDAS and OCSSW use the correct Python installation.
bcoll018 wrote: Tue Nov 09, 2021 1:59 pm America/New_York I am having this same problem with update_luts using SeaDAS 8.1.0 through the GUI, as well as through the command line prompt (Mac Big Sur 11.16). The workaround posted by daurin does not solve this problem for me. I can import "requests" using python3 and python, but it cannot be found when called by update_luts. Maybe it is calling python2 somehow?
See: https://docs.anaconda.com/anaconda/install/mac-os/ Your ~/.zshrc file should have the lines:

Code: Select all

source <path to conda>/bin/activate
conda init zsh
Since ocssw_runner uses bash, you need to add the following lines to your ~/.bashrc:

Code: Select all

source <path to conda>/bin/activate
conda init sh

ruman09
Posts: 1
Joined: Wed Nov 10, 2021 5:04 am America/New_York

Re: SeaDAS 8 install_OCSSW problem with python requests

by ruman09 » Wed Nov 10, 2021 5:15 am America/New_York

Download the Git bundles, README, OCSSW_env.bash, sha1sum file, and install script:
$ cd /tmp
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocss ... l_ocssw.py
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocssw/README
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocssw/OCSSW_bash.env
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocss ... .sha256sum
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocssw/scripts.bundle
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocssw/common.bundle
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocss ... _64.bundle
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocss ... _64.bundle
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocssw/ocrvc.bundle
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocssw/modis.bundle
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocssw/modisa.bundle
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocssw/hmodisa.bundle
$ wget -cN http://oceandata.sci.gsfc.nasa.gov/ocssw/seawifs.bundle
Make the install script executable, and run it:
$ chmod +x install_ocssw.py
$ ./install_ocssw.py –install-dir=<seadas-<version> install directory>/ocssw –git-branch=v<version> –local=/tmp –aqua –seawifs
Re-run the installer after it finishes the “local” install to get the updates added after the bundles were created:
sudo ./install_ocssw.py –install-dir=<seadas-<version> install directory>/ocssw –git-branch=v<version> –aqua –seawifs

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 200
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1

Re: SeaDAS 8 install_OCSSW problem with python requests

by OB SeaDAS - xuanyang02 » Wed Nov 10, 2021 10:43 am America/New_York

It seems that you have 'aqua', 'terra', 'seawifs', 'viirsn', 'viirsj1' installed, but have problems with update_luts?

SeaDAS points to /Users/bcollist/opt/anaconda3/bin/python3. Can you try

Code: Select all

/Users/bcollist/opt/anaconda3/bin/python3 -m pip install requests
You need Python requests package v2.18.0 or later

Post Reply