Slow download speeds for MISR data

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
joeyhotz
Posts: 4
Joined: Tue Aug 30, 2022 11:58 am America/New_York
Answers: 0

Slow download speeds for MISR data

by joeyhotz » Tue Aug 30, 2022 12:07 pm America/New_York

Hi all, I have been trying to use a script which I made in R to download MISR NetCDF files from the MIL2ASAE.003 dataset on the OPeNDAP server.

In my experience, I have found that the download speeds for these NetCDF files are quite quick at first, but after a certain point (around 50 files), it starts to take around 10 minutes to download a single NetCDF file.

I am downloading approximately 250 files for each year (from 2000-2021) in order to make annual datasets, and as it stands, it takes around 48 hours to download all the necessary NetCDF files to create each of these individual annual datasets.

Does anyone know of a workaround to make these file downloads quicker? I suspect that there might be some system which times me out after too many download requests in a short time period, but I have not been able to find any information on whether this is the case.

Tags:

ASDC - ingridgs
Subject Matter Expert
Subject Matter Expert
Posts: 142
Joined: Fri Apr 23, 2021 9:14 am America/New_York
Answers: 1
Has thanked: 17 times
Been thanked: 7 times

Re: Slow download speeds for MISR data

by ASDC - ingridgs » Tue Aug 30, 2022 1:06 pm America/New_York

Hello,

You can try to download data via Direct Data Download:
https://asdc.larc.nasa.gov/data/MISR/

Also, you can find here in the Forum scripts to download multiple files with Python and wget.
wget: viewtopic.php?t=2329
Python: viewtopic.php?t=2330

Regards,

joeyhotz
Posts: 4
Joined: Tue Aug 30, 2022 11:58 am America/New_York
Answers: 0

Re: Slow download speeds for MISR data

by joeyhotz » Tue Aug 30, 2022 3:04 pm America/New_York

Thanks for the reply, I'll try and edit my code to use wget in the script.

Do you know if using wget would help alleviate the issue which I have been having with slow downloads? I have been using the "download.file()" function in R instead of wget and I have noticed that it often takes around 10 minutes to download a single NetCDF file when running my script.

ASDC - ingridgs
Subject Matter Expert
Subject Matter Expert
Posts: 142
Joined: Fri Apr 23, 2021 9:14 am America/New_York
Answers: 1
Has thanked: 17 times
Been thanked: 7 times

Re: Slow download speeds for MISR data

by ASDC - ingridgs » Wed Aug 31, 2022 6:11 am America/New_York

Hi @joeyhotz

I have not used R to download multiple files from ASDC data, so I wouldn't know how to answer. However, I have tried both the wget and Python scripts, and I have always been able to retrieve multiple files within minutes.

Regards,

Post Reply