Downloading data from CDDIS archive.

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
anil20
Posts: 2
Joined: Wed Mar 15, 2023 9:16 am America/New_York
Answers: 0

Downloading data from CDDIS archive.

by anil20 » Wed Mar 15, 2023 9:29 am America/New_York

My code
proxy ={"https": f"http://{username}:{password}@{proxyserver}:{port}"}

url = 'https://cddis.nasa.gov/archive/gnss/data/daily/2017/003/17o/mate0030.17o.Z'

response = requests.get(url, proxies=proxy) auth=(A_username, A_password))

if response.status_code == 200:
# Check the Content-Length header to get the expected file size
expected_size = int(response.headers.get("Content-Length", 0))

# Write the downloaded content to a file
with open('mate0030.17o.Z', 'wb') as f:
for chunk in response.iter_content(chunk_size=1000):
if chunk:
f.write(chunk)
print(f"Downloaded {len(chunk)} bytes...")
# Check the actual size of the downloaded file
actual_size = os.path.getsize('mate0030.17o.Z')

# Compare the expected and actual sizes to ensure the file was fully downloaded
if actual_size == expected_size:
print("File downloaded successfully!")
else:
print("File download incomplete.")
else:
print("Unable to download file.")

output of the code
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 1000 bytes...
Downloaded 178 bytes...
File download incomplete.


I have queries as listed below
1)I am able to access the file with my python code, but my data down is going incomplete. why?
2)How should the content of .netrc file should be feeded to my code? Here I am feeding the user credentials manually as I have no idea of interfacing this details.
Please suggest.

Filters:

CDDIS_support_Lori
User Services
User Services
Posts: 82
Joined: Mon Sep 30, 2019 11:17 am America/New_York
Answers: 0
Been thanked: 1 time

Re: Downloading data from CDDIS archive.

by CDDIS_support_Lori » Wed Mar 22, 2023 10:25 am America/New_York

What may be happening is that the Python Requests library is looking for your .netrc (or _netrc on windows) file in your home directory. Since it does not find it there no credentials are passed, and you are redirected to the Earthdata Login page. What you are downloading is the HTML code for that web page. This problem is described in our FAQ page. (https://cddis.nasa.gov/About/CDDIS_File_Download_FAQ.html) under #14. The easiest solution would be to move your .netrc (or _netrc on windows) file and try again.

anil20
Posts: 2
Joined: Wed Mar 15, 2023 9:16 am America/New_York
Answers: 0

Re: Downloading data from CDDIS archive.

by anil20 » Thu Mar 23, 2023 1:22 am America/New_York

Dear Sir/Madam,
I am attaching the Screenshot of the location and permission of the
NETRC file which is named as _netrc.text. Is it correct format? Also I
am sharing the codefile, which is exactly like the code on CCDIS site
except the proxy details.
This code is running perfectly in linux system but not on windows. I
request you to look into and provide the solution.
Thanks,
Attachments
code file
code file
codefile.PNG (29.08 KiB) Not viewed yet
Location and permission for _netrc file
Location and permission for _netrc file
netrc location.PNG (72.62 KiB) Not viewed yet

nathan
Posts: 3
Joined: Thu May 27, 2021 2:52 pm America/New_York
Answers: 0

Re: Downloading data from CDDIS archive.

by nathan » Fri Mar 24, 2023 3:34 pm America/New_York

Hi,

I notice that it looks like you credential file is named '_netrc.txt', please try to rename it to simply: '_netrc'. If that is corrected and you are still having issues, please reply with the contents of the file that you are able to download. That could provide us clues as to what is going wrong.

Post Reply