ASDC Response: 401/Unauthorized when using python3

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
edwardgrys
Posts: 5
Joined: Thu Mar 17, 2022 7:16 am America/New_York

ASDC Response: 401/Unauthorized when using python3

by edwardgrys » Thu Mar 17, 2022 7:07 pm America/New_York

I am trying to download CALIPSO data from ASDC using a python. scripts based on the examples on this forum. My example code is very simple

import requests
session = requests.Session()
session.auth = (<my username>, <my password>)

url = 'https://asdc.larc.nasa.gov/data/CALIPSO/LID_L1-Standard-V4-10/2007/01/CAL_LID_L1-Standard-V4-10.2007-01-01T00-22-49ZN.hdf'
r1 = session.get(url)

This returns a 401 response (unauthorised). I have my username and password correct, as I am able to download a MODIS file using the same code

url = 'https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD08_D3/2020/001/MYD08_D3.A2020001.061.2020003014337.hdf'
r1 = session.request('get', url)

Is a status: 200 (success). I can download this file using the wget instructions, so this appears to be a problem with the python code, but given this is a basic example, I am not sure what the issue is. The library debug output is below (with some bit removed)

Many thanks
Ed

----

In[9]: session.get(url)
DEBUG:urllib3.connectionpool:Resetting dropped connection: asdc.larc.nasa.gov
send: b'GET /data/CALIPSO/LID_L1-Standard-V4-10/2007/01/CAL_LID_L1-Standard-V4-10.2007-01-01T00-22-49ZN.hdf HTTP/1.1\r\nHost: asdc.larc.nasa.gov\r\nUser-Agent: python-requests/2.27.1\r\nAccept-Encoding: gzip, deflate, br\r\nAccept: */*\r\nConnection: keep-alive\r\nCookie: ----------------\r\nAuthorization: Basic -------------------\r\n\r\n'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Date: Thu, 17 Mar 2022 22:55:32 GMT
header: Server: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips
header: Content-Length: 381
header: Content-Type: text/html; charset=iso-8859-1
header: Strict-Transport-Security: max-age=31536000;includeSubDomains;preload
DEBUG:urllib3.connectionpool:https://asdc.larc.nasa.gov:443 "GET /data/CALIPSO/LID_L1-Standard-V4-10/2007/01/CAL_LID_L1-Standard-V4-10.2007-01-01T00-22-49ZN.hdf HTTP/1.1" 401 381
Out[9]: <Response [401]>

Tags:

ASDC - cheyenne.e.land
User Services
User Services
Posts: 61
Joined: Mon Mar 22, 2021 3:55 pm America/New_York
Answers: 1

Re: ASDC Response: 401/Unauthorized when using python3

by ASDC - cheyenne.e.land » Fri Mar 18, 2022 3:51 pm America/New_York

Hello,

Just from looking at your code, you might have a typo. You say that the MODIS file works using the same code but for CALIPSO, it says that r1 = session.get(url) and for MODIS it says r1 = session.request('get', url). We have also notified a subject matter expert on your inquiry. Thank you.

Regards,
ASDC User Services

edwardgrys
Posts: 5
Joined: Thu Mar 17, 2022 7:16 am America/New_York

Re: ASDC Response: 401/Unauthorized when using python3

by edwardgrys » Fri Mar 18, 2022 4:18 pm America/New_York

Thanks for the quick reply! That was from me doing some experimenting, but unfortunately it doesn't fix the issue (I think one method is actually just calling the other)

ASDC - cheyenne.e.land
User Services
User Services
Posts: 61
Joined: Mon Mar 22, 2021 3:55 pm America/New_York
Answers: 1

Re: ASDC Response: 401/Unauthorized when using python3

by ASDC - cheyenne.e.land » Sun Mar 20, 2022 5:31 pm America/New_York

Hello,

Just wanted to make sure that wasn't the issue and again, a subject matter expert has been notified about your inquiry and will get back with you soon.

Regards,
ASDC User Services

edwardgrys
Posts: 5
Joined: Thu Mar 17, 2022 7:16 am America/New_York

Re: ASDC Response: 401/Unauthorized when using python3

by edwardgrys » Mon Mar 21, 2022 5:46 am America/New_York

For anyone else who finds this question, it appears that a solution is to make the call to 'get' without supplying any authentication information.

session = Session()
response = session.get(url)
with open(outputfilename, 'wb') as f:
f.write(response.content)

appears to work for me when downloading CALIPSO data. I assume only unauthenticated access is not the intention though, so this might be changed in the future.

njester
Posts: 13
Joined: Sat Mar 06, 2021 9:03 am America/New_York

Re: ASDC Response: 401/Unauthorized when using python3

by njester » Mon Mar 21, 2022 8:59 am America/New_York

edwardgrys wrote: Mon Mar 21, 2022 5:46 am America/New_York For anyone else who finds this question, it appears that a solution is to make the call to 'get' without supplying any authentication information.

session = Session()
response = session.get(url)
with open(outputfilename, 'wb') as f:
f.write(response.content)

appears to work for me when downloading CALIPSO data. I assume only unauthenticated access is not the intention though, so this might be changed in the future.
You may want to verify that it downloaded what you think it downloaded. If you download without passing credentials, you probably downloaded a redirect page asking you to log in. Recently, systems were updated to require a token for login instead of credentials and I'm still in the process of testing and updating example scripts. You can find an updated Python example that can be used to download part a campaign here:
viewtopic.php?t=2330&sid=51ba8fe08ac1b2fc40bec7a8eee35a16
The script was just recently udpated, so if you have any issues with the script or forum post, let me know.

edwardgrys
Posts: 5
Joined: Thu Mar 17, 2022 7:16 am America/New_York

Re: ASDC Response: 401/Unauthorized when using python3

by edwardgrys » Mon Mar 21, 2022 10:29 am America/New_York

Thanks for the quick response - I had originally tried using an Earthdata token (I have a working script for downloading MODIS data from LAADS), but it didn't seem to work for downloading the CALIPSO data.

Good point about the download - I checked though and the files contain what looks like the correct lidar data. I am happy to send over the actual requests made if that is useful.

Post Reply