Unable to get ancillary data/luts using scripts (or, this old guy needs help)

[blesht]

Hi - It's been a while since I've done any SeaDAS processing, certainly not since the new authentication requirement has been operational.  I wasn't _that_ surprised when my attempt to update my ancillary data files failed with the following:

bash-3.2$ ./getanc.py -s 1997267000000 -e 2002365235959
Traceback (most recent call last):
  File "./getanc.py", line 10, in <module>
    import modules.anc_utils as ga
  File "/Applications/seadas-7.5/ocssw/scripts/modules/anc_utils.py", line 9, in <module>
    import modules.ProcUtils as ProcUtils
  File "/Applications/seadas-7.5/ocssw/scripts/modules/ProcUtils.py", line 16, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

I looked through the forum archive and, after trying to de-cypher some the advice, did the following:

1.  Created a ~/.netrc file that consists of one line:  machine urs.earthdata.nasa.gov login MYID password MYPW
2.  Installed the python requests library (used 'pipenv install requests')

I tried getanc.py again and got the same error message.  I also get an error message when I try update_luts.py.  That message is:

bash-3.2$ ./update_luts.py
Traceback (most recent call last):
  File "./update_luts.py", line 9, in <module>
    import modules.LutUtils as Lut
  File "/Applications/seadas-7.5/ocssw/scripts/modules/LutUtils.py", line 5, in <module>
    import JsonUtils as Session
ModuleNotFoundError: No module named 'JsonUtils'

Finally, when trying from the SeaDAS7.5.3 GUI to Update Data Processors, I get:

execution exception: java.io.IOException: install_ocssw.py failed with exit code 1.
Check log for more details.

.... Whole bunch of information .... followed by:

2020-03-15 19:32:29 (722 KB/s) - ‘opt-src-v7.5.tar’ saved [230021120/230021120]

Installing scripts (16 of 20)
Python version acceptable
Installing seawifs-luts (17 of 20)
Traceback (most recent call last):
  File "/Applications/seadas-7.5/ocssw/scripts/update_luts.py", line 9, in <module>
    import modules.LutUtils as Lut
  File "/Applications/seadas-7.5/ocssw/scripts/modules/LutUtils.py", line 5, in <module>
    import JsonUtils as Session
  File "/Applications/seadas-7.5/ocssw/scripts/modules/JsonUtils.py", line 7, in <module>
    from ProcUtils import getSession, httpdl
  File "/Applications/seadas-7.5/ocssw/scripts/modules/ProcUtils.py", line 16, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
Error - Could not install luts for seawifs

So - maybe I messed up installing the python requests library; are there some specific instructions somewhere?  I run SeaDAS on a MacBookPro under OS10.14.6

Thanks, Barry

[gnwiii]

It would be helpful if you can provide more details of your Python setup.   Are you using pipenv to create a virtual environment for the OCSSW Processing System?   The virtual environment page provides an example using requests.  Does this example work for you?   Can you provide the results from which python and python --version?  Are you running the SeaDAS GUI from the bash command-line or from the MacOS Applications menu?  Have you tried running the GUI both ways?

[blesht]

Hi George!  How's that retirement going?  :-)

Thanks for the suggestions.  I will be giving them a try (currently on an interminable conference call) shortly.  I'm not really (at all) python savvy, so have to check that link you provided.  I doubt that I am creating a virtual environment; I certainly haven't done anything manually to do that.  I can however, answer a few of your other questions now.

bash-3.2$ which python
/opt/local/bin/python
bash-3.2$ python --version
Python 3.7.7

I first encountered this problem when trying to generate some L2 files using l2gen from a long-used bash script.  The part of my script that calls getanc.py was failing.  I checked the forum and found some advice from Sean that suggested I needed to update my python scripts, so I just tried the OCSSW Update from the GUI.  That failed as noted.  I usually run the SeaDAS GUI from the Applications Menu (actually from the task bar icon).  I can easily try to run the GUI from the command line (if this call ever ends).

Cheers, Barry

[blesht]

OK - call finally over.  I'm afraid that I'm going to need more specific guidance on setting up a virtual environment for python.  I'm surprised that if this is now required for running SeaDAS there isn't something like a detailed instruction set already available - or maybe I just missed it.  Here is what I've done trying to follow the example in the link you provided - note that I use Macports to manage this kind of stuff.

1.  bash-3.2$ sudo port install py36-pip
--->  Computing dependencies for py36-pip
The following dependencies will be installed:
py36-setuptools
python36
Continue? [Y/n]: Y

2.  bash-3.2$ sudo port select --set pip pip36
Selecting 'pip36' for 'pip' succeeded. 'pip36' is now active.
bash-3.2$ which pip
/opt/local/bin/pip
bash-3.2$ pip --version
pip 20.0.2 from /opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip (python 3.6)

3.  bash-3.2$ pip install --user pipenv
Requirement already satisfied: pipenv in ./Library/Python/3.6/lib/python/site-packages (2018.11.26)
Requirement already satisfied: virtualenv-clone>=0.2.5 in ./Library/Python/3.6/lib/python/site-packages (from pipenv) (0.5.3)
Requirement already satisfied: virtualenv in ./Library/Python/3.6/lib/python/site-packages (from pipenv) (20.0.10)
Requirement already satisfied: certifi in ./Library/Python/3.6/lib/python/site-packages (from pipenv) (2019.11.28)
Requirement already satisfied: setuptools>=36.2.1 in /opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from pipenv) (43.0.0)
Requirement already satisfied: pip>=9.0.1 in /opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from pipenv) (20.0.2)
Requirement already satisfied: importlib-resources<2,>=1.0; python_version < "3.7" in ./Library/Python/3.6/lib/python/site-packages (from virtualenv->pipenv) (1.3.1)
Requirement already satisfied: appdirs<2,>=1.4.3 in ./Library/Python/3.6/lib/python/site-packages (from virtualenv->pipenv) (1.4.3)
Requirement already satisfied: distlib<1,>=0.3.0 in ./Library/Python/3.6/lib/python/site-packages (from virtualenv->pipenv) (0.3.0)
Requirement already satisfied: six<2,>=1.9.0 in ./Library/Python/3.6/lib/python/site-packages (from virtualenv->pipenv) (1.14.0)
Requirement already satisfied: filelock<4,>=3.0.0 in ./Library/Python/3.6/lib/python/site-packages (from virtualenv->pipenv) (3.0.12)
Requirement already satisfied: importlib-metadata<2,>=0.12; python_version < "3.8" in ./Library/Python/3.6/lib/python/site-packages (from virtualenv->pipenv) (1.5.0)
Requirement already satisfied: zipp>=0.4; python_version < "3.8" in ./Library/Python/3.6/lib/python/site-packages (from importlib-resources<2,>=1.0; python_version < "3.7"->virtualenv->pipenv) (3.1.0)

I set up a new folder called 'project_folder' and then:

4. bash-3.2$ cd project_folder
    bash-3.2$ pipenv install requests

This results in some garbled output among which are the lines "To activate this projects's virtualenv, run pipenv shell." and "Alternatively, run a command inside the virtualenv with pipenv run."

Forging ahead, I created a 'main.py' file as listed and

5. bash-3.2$ cd Desktop/project_folder/
bash-3.2$ ls
main.py
bash-3.2$ cat main.py
import requests
response = requests.get('https://httpbin.org/ip')
print('Your IP is {0}'.format(response.json()['origin']))
bash-3.2$ pipenv run python main.py
Your IP is xx.xxx.xx.xx

Which seems to work.

So how does this help?  I did try again to run update_luts.py and getanc.py and get the same errors as before.  I also tried running pipenv run python main update_luts.py (prepending the SeaDAS ocssw/scripts directory name) and get the same error.  Surely, the solution can't be this involved. 

Anyway, would be grateful for any addition insight/suggestions.

Barry

[gnwiii]

I was enjoying retirement until COVID-19 came along, but now that we "live in interesting times" I'm dusting off old epidemiology references.  

I think pipenv creates a python virtual environment.  If you didn't intend to use a virtual environment, and seeing that you are using python from macports, sudo port install py37-requests should get the OCSSW downloads going (works for me on El Capitan).

[blesht]

George - I'm sure I've said this before, but I really appreciate all the help I've gotten from you over the years.  Your last suggestion worked perfectly and all seems well with respect to updates, getting ancillary data, etc. 

Stay safe and healthy and many, many thanks.

Barry