auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
pwurster_usgs
Posts: 3
Joined: Fri Mar 03, 2023 1:34 pm America/New_York
Answers: 0

auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

by pwurster_usgs » Fri Mar 10, 2023 11:33 am America/New_York

I have a Python script that used to download NLDAS netcdf files via "hydro1.gesdisc.eosdis.nasa.gov", however the script now returns a 401 Error (unauthorized). I cannot say when the change from successful download to 401 Error occurred, but sometime in the last 30-40 days or so. The script was according to the second example given by Earthdata at https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+Python

I have also tried the third example in the latter url, and also the example given under "Python using 'Requests'" example given at https://disc.gsfc.nasa.gov/data-access#python-requests
both of which return 401 Error.

I am running these scripts on a remote high performance computer via SLURM job, which makes Python 'Requests' methods ideal rather than using wget.

I can download these data fine using my browser on the local machine after logging into Earthdata.

What can I do to provide the correct credentials or get around the 401 Error so I can automatically download these data.

Tags:

GES DISC - mgreene
User Services
User Services
Posts: 191
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

by GES DISC - mgreene » Fri Mar 10, 2023 5:11 pm America/New_York

This is the GES DISC User Service, your post have been forwarded to an Subject Matter Expert for assistance.

GES DISC - mgreene
User Services
User Services
Posts: 191
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

by GES DISC - mgreene » Mon Mar 13, 2023 4:41 pm America/New_York

May I suggest to check that you have upgraded Python and requests module. Also check content and permissions of .netrc file.

pwurster_usgs
Posts: 3
Joined: Fri Mar 03, 2023 1:34 pm America/New_York
Answers: 0

Re: auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

by pwurster_usgs » Mon Mar 13, 2023 8:49 pm America/New_York

Python is 3.9.7

In Requests, I authorize with username and password arguments and not a .netrc

Ideally, anyone with Earthdata username and password can run the code to retrieve the data without .netrc

GES DISC - mgreene
User Services
User Services
Posts: 191
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

by GES DISC - mgreene » Wed Mar 15, 2023 9:09 am America/New_York

from requests.auth import HTTPDigestAuth
URL = 'your_URL_string_goes_here'
requests.get(URL, auth=HTTPDigestAuth('your_username', 'your_password'))

pwurster_usgs
Posts: 3
Joined: Fri Mar 03, 2023 1:34 pm America/New_York
Answers: 0

Re: auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

by pwurster_usgs » Wed Mar 15, 2023 9:40 am America/New_York

Still get 401 Error when using

from requests.auth import HTTPDigestAuth
URL = 'your_URL_string_goes_here'
requests.get(URL, auth=HTTPDigestAuth('your_username', 'your_password'))

I changed my Earthdata password (no interfering characters) and verified login via browser, I am sure the username and password are correct in the code. I have a personal Earthdata account associated with university, I will try with that account (later today) and report back. It is odd that methods similar to the above worked fine in January. Something changed and I do not think it is client side.

Here is a reference url of data I would like to collect

https://hydro1.gesdisc.eosdis.nasa.gov/data/NLDAS/NLDAS_VIC0125_M.2.0/2017/NLDAS_VIC0125_M.A201708.020.nc

The link downloads the data fine using the browser.

cbattisto
Posts: 11
Joined: Wed Feb 16, 2022 4:38 pm America/New_York
Answers: 0

Re: auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

by cbattisto » Mon Apr 17, 2023 11:42 am America/New_York

Hello,

Recently, the HTTPDigestAuth method of passing in your EDL credentials without a netrc have caused some inconsistent results. Please follow the steps below to create a netrc, and then test again using requests():

1. Create your netrc file (as well as your .urs_cookies file) by following this how-to: https://disc.gsfc.nasa.gov/information/howto?title=How%20to%20Generate%20Earthdata%20Prerequisite%20Files

2. Use the following code to download the granule:
https://disc.gsfc.nasa.gov/data-access#python-requests

Thanks for your patience!

GES DISC - mgreene
User Services
User Services
Posts: 191
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

by GES DISC - mgreene » Tue Apr 25, 2023 10:29 am America/New_York

Following up to see if the information provide met your needs. thanks

Post Reply