504 gateway timeout downloading MODIS l1b data

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
csaundersshultz
Posts: 3
Joined: Fri Aug 26, 2022 7:43 pm America/New_York
Answers: 0

504 gateway timeout downloading MODIS l1b data

by csaundersshultz » Fri Aug 26, 2022 8:10 pm America/New_York

Hello,
I am downloading some MODIS l1b data and consistently getting a 504 gateway timeout error when I use the "download script" option, but only after it successfully downloads anywhere from 50-200 files. When I try using 'wget --username=myusername --password=mypassword -i download.txt' it only downloads the html version of the page, not the actual file. Is anyone familiar with this issue and how to resolve it?

Tags:

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: 504 gateway timeout downloading MODIS l1b data

by LAADS_UserServices_M » Mon Aug 29, 2022 2:10 pm America/New_York

Send your wget command (without token) for us to investigate the issue?
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

csaundersshultz
Posts: 3
Joined: Fri Aug 26, 2022 7:43 pm America/New_York
Answers: 0

Re: 504 gateway timeout downloading MODIS l1b data

by csaundersshultz » Mon Aug 29, 2022 3:15 pm America/New_York

The wget command I am trying is:
wget --username=myusername --password=mypassword -i download.txt
where download.txt is just a textfile with a url on each line. The first two lines are:
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/2022/006/MOD021KM.A2022006.2325.006.2022007122956.hdf
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/2022/006/MOD021KM.A2022006.2150.006.2022007122912.hdf

This leads to a 13kb file being downloaded, which I am pretty sure is the html file of the link and not the actual .hdf file at the url which should be ~100mb. The wget command also does not work when I try to do a single file, e.g.:
wget --username=myusername --password=mypassword https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/2022/006/MOD021KM.A2022006.2325.006.2022007122956.hdf
Last edited by csaundersshultz on Mon Aug 29, 2022 3:16 pm America/New_York, edited 1 time in total.

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: 504 gateway timeout downloading MODIS l1b data

by LAADS_UserServices_M » Tue Aug 30, 2022 8:28 am America/New_York

The LAADS wget command syntax is like:
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/PATH_TO_DATA_DIRECTORY" --header "Authorization: Bearer MY_TOKEN" -P TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM

where:
PATH_TO_DATA_DIRECTORY: location of source directory in LAADS Archive
TOKEN: Your token from ladsweb
TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM: Where you would like to download the files. Examples include /Users/jdoe/data for macOS and Linux or C:\Users\jdoe\data for Windows


If like you can try wget script provided by LAADS by using following steps:
1. Go to data folder, for example:
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/
2. "Login" at the top of the page.
3. Click on "See wget Download Command" tab, you will see complete wget command with your token already included. Try this command to download data. Or, you can see examples of other download scripts
by clicking on "Download Help" tab.
Just curious that you are downloading Collection version 6 data whereas the latest version is 6.1. See:
https://ladsweb.modaps.eosdis.nasa.gov (/alerts-and-issues/90903)
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

csaundersshultz
Posts: 3
Joined: Fri Aug 26, 2022 7:43 pm America/New_York
Answers: 0

Re: 504 gateway timeout downloading MODIS l1b data

by csaundersshultz » Tue Aug 30, 2022 12:48 pm America/New_York

@LaadsUserServices

Thanks, I will try that WGET command and update here if it works. I used the earthdata search tool to select this data, so I guess earthdata does not point to the most updated version. Is it possible that some of the files are just missing? The download seems to 504 timeout error on the same files every time.

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: 504 gateway timeout downloading MODIS l1b data

by LAADS_UserServices_M » Wed Aug 31, 2022 7:49 am America/New_York

Both Collections are available and are listed in earthdata search tool. It is user's choice which one is selected. All files are available except where there is a data outage.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

csaundersshultz
Posts: 3
Joined: Fri Aug 26, 2022 7:43 pm America/New_York
Answers: 0

Re: 504 gateway timeout downloading MODIS l1b data

by csaundersshultz » Fri Sep 02, 2022 1:32 pm America/New_York

Okay I finally found the solution I needed, posting here in case others find it useful. I ended up using the Wget command which is given from the LAADS DAAC (which stores the data but only allows queries up to 2000 files) in combination with filenames given from an Earthdata query which doesn't have a size limit. This looked like this:

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 -i file_urls.txt --header "Authorization: Bearer PERSONAL_TOKEN_login_to_LAADS_to_get_this" -P .

Post Reply