Error downloading SST data with obdaac_download.py

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
larteaga
Posts: 9
Joined: Fri Sep 22, 2017 3:28 pm America/New_York
Answers: 0

Error downloading SST data with obdaac_download.py

by larteaga » Mon Apr 05, 2021 1:33 pm America/New_York

Hello,

I am trying to download a list of monthly MODIS-Aqua SST files by reading a .txt file. When I run:
python obdaac_download.py --filelist SST_monthly_2002_2020_list.txt

I get this output for each file (no file is downloaded):
Whoops...problem retrieving https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20201201_20201231.L3b.MO.SST.nc (received status 401)

A sample of the list:
https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20020701_20020731.L3b.MO.SST.nc
https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20020801_20020831.L3b.MO.SST.nc
https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20020901_20020930.L3b.MO.SST.nc
https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20021001_20021031.L3b.MO.SST.nc

Thank you in advance for any help.
Cheers,
Lionel

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Re: Error downloading SST data with obdaac_download.py

by OB.DAAC - SeanBailey » Mon Apr 05, 2021 1:44 pm America/New_York

Lionel,

Your list worked for me (BTW, while it works as you have it, you don't need the full URL, just the filename will do). If you run obdaac_download.py with the verbose option ( -v or even -vv), there may be a clue that will help diagnose the problem you're having.

Sean

larteaga
Posts: 9
Joined: Fri Sep 22, 2017 3:28 pm America/New_York
Answers: 0

Re: Error downloading SST data with obdaac_download.py

by larteaga » Mon Apr 05, 2021 3:18 pm America/New_York

Thank you, Sean. I can't figure out why it does not work for me. Adding -v just prints "Retrieving AQUA_MODIS.20201201_20201231.L3b.MO.SST.nc" before the error. I've tried in two different computers..

Lionel

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Re: Error downloading SST data with obdaac_download.py

by OB.DAAC - SeanBailey » Mon Apr 05, 2021 3:59 pm America/New_York

Add two v's (-vv) and you should see some debug information, e.g.:

obdaac_download -vv --filelist=listfile.txt
Output directory: /ssdwork/junk
Retrieving AQUA_MODIS.20201201_20201231.L3b.MO.SST.nc
DEBUG:urllib3.util.retry:Converted retries value: 5 -> Retry(total=5, connect=None, read=None, redirect=None, status=None)
OBPG session started
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oceandata.sci.gsfc.nasa.gov
DEBUG:urllib3.connectionpool:https://oceandata.sci.gsfc.nasa.gov:443 "GET /ob/getfile/AQUA_MODIS.20201201_20201231.L3b.MO.SST.nc HTTP/1.1" 302 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): urs.earthdata.nasa.gov
DEBUG:urllib3.connectionpool:https://urs.earthdata.nasa.gov:443 "GET /oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA HTTP/1.1" 302 None
DEBUG:urllib3.connectionpool:https://oceandata.sci.gsfc.nasa.gov:443 "GET /ob/getfile/restrict?code=ec5d844611bd2bcb3ec97fab7ad2b2408215f9c3cc79e28693486983cbe2aa5f HTTP/1.1" 302 None
DEBUG:urllib3.connectionpool:https://oceandata.sci.gsfc.nasa.gov:443 "GET /ob/getfile/AQUA_MODIS.20201201_20201231.L3b.MO.SST.nc HTTP/1.1" 200 259094038
Downloading AQUA_MODIS.20201201_20201231.L3b.MO.SST.nc ( 247.09 MBs)
[==================================================]...Done

larteaga
Posts: 9
Joined: Fri Sep 22, 2017 3:28 pm America/New_York
Answers: 0

Re: Error downloading SST data with obdaac_download.py

by larteaga » Mon Apr 05, 2021 4:34 pm America/New_York

Thanks. I'm not much of a python expert, but this is what I get with -vv:
Output directory: /home/larteaga
Retrieving https://oceandata.sci.gsfc.nasa.gov/cgi ... .MO.SST.nc
Session started
DEBUG:urllib3.util.retry:Converted retries value: 5 -> Retry(total=5, connect=None, read=None, redirect=None, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oceandata.sci.gsfc.nasa.gov:443
DEBUG:urllib3.connectionpool:https://oceandata.sci.gsfc.nasa.gov:443 "GET /cgi/getfile/AQUA_MODIS.20020701_20020731.L3b.MO.SST.nc HTTP/1.1" 302 None
DEBUG:urllib3.connectionpool:https://oceandata.sci.gsfc.nasa.gov:443 "GET /ob/getfile/AQUA_MODIS.20020701_20020731.L3b.MO.SST.nc HTTP/1.1" 302 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): urs.earthdata.nasa.gov:443
DEBUG:urllib3.connectionpool:https://urs.earthdata.nasa.gov:443 "GET /oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA HTTP/1.1" 200 None
Whoops...problem retrieving https://oceandata.sci.gsfc.nasa.gov/cgi ... .MO.SST.nc (received status 401).

Curiously, the download works for me on a third computer that has python3.7.9, while the other two, where it fails, have python3.8.5. That's the only difference that I have spotted

Cheers,
Lionel

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Re: Error downloading SST data with obdaac_download.py

by OB.DAAC - SeanBailey » Tue Apr 06, 2021 1:55 pm America/New_York

Lionel,
Sorry I missed this in the original post...the error code 401. That suggests that your Earthdata Login credentials in your .netrc file are not correct. If you can confirm that they are identical to the one on the machine that does work, then we'll have to scratch our heads some more, because I'm not seeing what else the problem could be...

Sean

Post Reply