Downloading ASDC Data with Python 3

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
alphayash
Posts: 1
Joined: Thu Jul 21, 2022 11:18 pm America/New_York
Answers: 0

Re: Downloading ASDC Data with Python 3

by alphayash » Thu Jul 21, 2022 11:19 pm America/New_York

The easiest way to download and save a file is to use the urllib.request.urlretrieve function:

import urllib.request
...
# Download the file from `url` and save it locally under `file_name`:
urllib.request.urlretrieve(url, file_name).
import urllib.request
...
# Download the file from `url`, save it in a temporary directory and get the
# path to it (e.g. '/tmp/tmpb48zma.txt') in the `file_name` variable:
file_name, headers = urllib.request.urlretrieve(url)
But keep in mind that urlretrieve is considered legacy and might become deprecated (not sure why, though).[url]https://vidmateapp.win/homepage/[/url]

So the most correct way to do this would be to use the urllib.request.urlopen function to return a file-like object that represents an HTTP response and copy it to a real file using shutil.copyfileobj.

import urllib.request
import shutil.
...
# Download the file from `url` and save it locally under `file_name`:
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
If this seems too complicated, you may want to go simpler and store the whole download in a bytes object and then write it to a file. But this works well only for small files.

import urllib.request
...
# Download the file from `url` and save it locally under `file_name`:
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
data = response.read() # a `bytes` object
out_file.write(data)
It is possible to extract .gz (and maybe other formats) compressed data on the fly, but such an operation probably requires the HTTP server to support random access to the file.

import urllib.request
import gzip
...
# Read the first 64 bytes of the file inside the .gz archive located at `url`
url = '(http://) example.com/something.gz'
with urllib.request.urlopen(url) as response:
with gzip.GzipFile(fileobj=response) as uncompressed:
file_header = uncompressed.read(64) # a `bytes` object
# Or do anything shown above using `uncompressed` instead of `response`.
Last edited by alphayash on Mon Oct 10, 2022 2:27 pm America/New_York, edited 3 times in total.

Tags:

njester
Posts: 20
Joined: Sat Mar 06, 2021 9:03 am America/New_York
Answers: 0
Been thanked: 3 times

Re: Downloading ASDC Data with Python 3

by njester » Fri Jul 22, 2022 10:03 am America/New_York

Those are great examples of how to download without using the requests library. Anyone using this example should remember that they'll need to include the token in the header to authenticate with Earthdata Login.

Post Reply