Downloading MOD13A1 and MYD13A1 files - methods?

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
binxu748er
Posts: 3
Joined: Mon Feb 19, 2024 11:31 am America/New_York
Answers: 0

Downloading MOD13A1 and MYD13A1 files - methods?

by binxu748er » Mon Feb 19, 2024 11:41 am America/New_York

Hello everyone, I'm very new to this site and need it for my research.

I seem to have the problem, when downloading all the files at once using a python code from the saved file.txt with the links to the files.hdf, all downloaded files turn out to be "corrupted"/unusable. However, if I download each file manually, the files seems to be fine.

Same problem with the cygwin method.

Is there a quicker way to download the files? I have to download roughly 2000 files

Many thanks!

Tags:

LP DAAC - lien
User Services
User Services
Posts: 182
Joined: Thu Jun 25, 2020 9:51 am America/New_York
Answers: 0
Been thanked: 4 times

Re: Downloading MOD13A1 and MYD13A1 files - methods?

by LP DAAC - lien » Tue Feb 20, 2024 9:29 am America/New_York

Hello,
Could you attach your python code and we can take a look, Also, could you attach the text file then we can see where you compile it from. If you would like you could send it to the LPDAAC email: lpdaac@usgs.gov
Thanks,
Brett

binxu748er
Posts: 3
Joined: Mon Feb 19, 2024 11:31 am America/New_York
Answers: 0

Re: Downloading MOD13A1 and MYD13A1 files - methods?

by binxu748er » Tue Feb 20, 2024 2:48 pm America/New_York

Hello,
Py code:

import requests
from tqdm import tqdm

def download_hdf_files(links_file, download_path):
with open(links_file, 'r') as file:
links = file.readlines()

for link in tqdm(links, desc="Downloading files...", unit="file"):
link = link.strip()
filename = link.split("/")[-1]
filepath = f"{download_path}/{filename}"


response = requests.get(link, stream=True)
total_size = int(response.headers.get('content-length', 0))

with open(filepath, 'wb') as file, tqdm(
desc=filename,
total=total_size,
unit='B',
unit_scale=True,
unit_divisor=1024,
) as bar:
for data in response.iter_content(chunk_size=1024):
bar.update(len(data))
file.write(data)

if __name__ == "__main__":
#Textfile
links_file_path = "C:/folder/file.txt..."

#target folder
download_directory = "C:/folder2/..."

download_hdf_files(links_file_path, download_directory)


Text file (converted to PDF for upload purpose) is attached. I have only uploaded one txt (PDF) file for MOD13A1 (549 entries). The procedure for MYD13A1 and other files with albedo data is the same.
Attachments
mod13a1raw-1.pdf
(146.65 KiB) Downloaded 3794 times

LP DAAC - dgolon
User Services
User Services
Posts: 292
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 18 times
Been thanked: 2 times
Contact:

Re: Downloading MOD13A1 and MYD13A1 files - methods?

by LP DAAC - dgolon » Thu Feb 22, 2024 10:06 am America/New_York

Hello @binxu748er I spoke with our science team and they recommended using the wget to bulkdown the data. You can find more information here: https://github.com/nasa/LPDAAC-Data-Resources/blob/main/guides/bulk_download_using_wget.md Thank you -- Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

binxu748er
Posts: 3
Joined: Mon Feb 19, 2024 11:31 am America/New_York
Answers: 0

Re: Downloading MOD13A1 and MYD13A1 files - methods?

by binxu748er » Thu Feb 22, 2024 10:53 am America/New_York

Hello, it worked now. Many thanks for the support.

Post Reply