auto-download NLDAS VIC from hydro1.gesdisc.eosdis.nasa.gov using Python "Requests"

[pwurster_usgs]

I have a Python script that used to download NLDAS netcdf files via "hydro1.gesdisc.eosdis.nasa.gov", however the script now returns a 401 Error (unauthorized). I cannot say when the change from successful download to 401 Error occurred, but sometime in the last 30-40 days or so. The script was according to the second example given by Earthdata at https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+Python

I have also tried the third example in the latter url, and also the example given under "Python using 'Requests'" example given at https://disc.gsfc.nasa.gov/data-access#python-requests
both of which return 401 Error.

I am running these scripts on a remote high performance computer via SLURM job, which makes Python 'Requests' methods ideal rather than using wget.

I can download these data fine using my browser on the local machine after logging into Earthdata.

What can I do to provide the correct credentials or get around the 401 Error so I can automatically download these data.

[mgreene]

This is the GES DISC User Service, your post have been forwarded to an Subject Matter Expert for assistance.

[mgreene]

May I suggest to check that you have upgraded Python and requests module. Also check content and permissions of .netrc file.

[pwurster_usgs]

Python is 3.9.7

In Requests, I authorize with username and password arguments and not a .netrc

Ideally, anyone with Earthdata username and password can run the code to retrieve the data without .netrc

[mgreene]

from requests.auth import HTTPDigestAuth
URL = 'your_URL_string_goes_here'
requests.get(URL, auth=HTTPDigestAuth('your_username', 'your_password'))

[pwurster_usgs]

Still get 401 Error when using

from requests.auth import HTTPDigestAuth
URL = 'your_URL_string_goes_here'
requests.get(URL, auth=HTTPDigestAuth('your_username', 'your_password'))

I changed my Earthdata password (no interfering characters) and verified login via browser, I am sure the username and password are correct in the code. I have a personal Earthdata account associated with university, I will try with that account (later today) and report back. It is odd that methods similar to the above worked fine in January. Something changed and I do not think it is client side.

Here is a reference url of data I would like to collect

https://hydro1.gesdisc.eosdis.nasa.gov/data/NLDAS/NLDAS_VIC0125_M.2.0/2017/NLDAS_VIC0125_M.A201708.020.nc

The link downloads the data fine using the browser.

[GES DISC - cbattisto]

Hello,

Recently, the HTTPDigestAuth method of passing in your EDL credentials without a netrc have caused some inconsistent results. Please follow the steps below to create a netrc, and then test again using requests():

1. Create your netrc file (as well as your .urs_cookies file) by following this how-to: https://disc.gsfc.nasa.gov/information/howto?title=How%20to%20Generate%20Earthdata%20Prerequisite%20Files

2. Use the following code to download the granule:
https://disc.gsfc.nasa.gov/data-access#python-requests

Thanks for your patience!

[mgreene]

Following up to see if the information provide met your needs. thanks