504 gateway timeout downloading MODIS l1b data
-
- Posts: 3
- Joined: Fri Aug 26, 2022 7:43 pm America/New_York
504 gateway timeout downloading MODIS l1b data
Hello,
I am downloading some MODIS l1b data and consistently getting a 504 gateway timeout error when I use the "download script" option, but only after it successfully downloads anywhere from 50-200 files. When I try using 'wget --username=myusername --password=mypassword -i download.txt' it only downloads the html version of the page, not the actual file. Is anyone familiar with this issue and how to resolve it?
I am downloading some MODIS l1b data and consistently getting a 504 gateway timeout error when I use the "download script" option, but only after it successfully downloads anywhere from 50-200 files. When I try using 'wget --username=myusername --password=mypassword -i download.txt' it only downloads the html version of the page, not the actual file. Is anyone familiar with this issue and how to resolve it?
Tags:
-
- User Services
- Posts: 285
- Joined: Mon Sep 30, 2019 8:33 am America/New_York
- Has thanked: 3 times
Re: 504 gateway timeout downloading MODIS l1b data
Send your wget command (without token) for us to investigate the issue?
Regards,
LAADS User Services
To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.
LAADS User Services
To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.
-
- Posts: 3
- Joined: Fri Aug 26, 2022 7:43 pm America/New_York
Re: 504 gateway timeout downloading MODIS l1b data
The wget command I am trying is:
wget --username=myusername --password=mypassword -i download.txt
where download.txt is just a textfile with a url on each line. The first two lines are:
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/2022/006/MOD021KM.A2022006.2325.006.2022007122956.hdf
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/2022/006/MOD021KM.A2022006.2150.006.2022007122912.hdf
This leads to a 13kb file being downloaded, which I am pretty sure is the html file of the link and not the actual .hdf file at the url which should be ~100mb. The wget command also does not work when I try to do a single file, e.g.:
wget --username=myusername --password=mypassword https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/2022/006/MOD021KM.A2022006.2325.006.2022007122956.hdf
wget --username=myusername --password=mypassword -i download.txt
where download.txt is just a textfile with a url on each line. The first two lines are:
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/2022/006/MOD021KM.A2022006.2325.006.2022007122956.hdf
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/2022/006/MOD021KM.A2022006.2150.006.2022007122912.hdf
This leads to a 13kb file being downloaded, which I am pretty sure is the html file of the link and not the actual .hdf file at the url which should be ~100mb. The wget command also does not work when I try to do a single file, e.g.:
wget --username=myusername --password=mypassword https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/2022/006/MOD021KM.A2022006.2325.006.2022007122956.hdf
Last edited by csaundersshultz on Mon Aug 29, 2022 3:16 pm America/New_York, edited 1 time in total.
-
- User Services
- Posts: 285
- Joined: Mon Sep 30, 2019 8:33 am America/New_York
- Has thanked: 3 times
Re: 504 gateway timeout downloading MODIS l1b data
The LAADS wget command syntax is like:
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/PATH_TO_DATA_DIRECTORY" --header "Authorization: Bearer MY_TOKEN" -P TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM
where:
PATH_TO_DATA_DIRECTORY: location of source directory in LAADS Archive
TOKEN: Your token from ladsweb
TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM: Where you would like to download the files. Examples include /Users/jdoe/data for macOS and Linux or C:\Users\jdoe\data for Windows
If like you can try wget script provided by LAADS by using following steps:
1. Go to data folder, for example:
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/
2. "Login" at the top of the page.
3. Click on "See wget Download Command" tab, you will see complete wget command with your token already included. Try this command to download data. Or, you can see examples of other download scripts
by clicking on "Download Help" tab.
Just curious that you are downloading Collection version 6 data whereas the latest version is 6.1. See:
https://ladsweb.modaps.eosdis.nasa.gov (/alerts-and-issues/90903)
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/PATH_TO_DATA_DIRECTORY" --header "Authorization: Bearer MY_TOKEN" -P TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM
where:
PATH_TO_DATA_DIRECTORY: location of source directory in LAADS Archive
TOKEN: Your token from ladsweb
TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM: Where you would like to download the files. Examples include /Users/jdoe/data for macOS and Linux or C:\Users\jdoe\data for Windows
If like you can try wget script provided by LAADS by using following steps:
1. Go to data folder, for example:
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD021KM/
2. "Login" at the top of the page.
3. Click on "See wget Download Command" tab, you will see complete wget command with your token already included. Try this command to download data. Or, you can see examples of other download scripts
by clicking on "Download Help" tab.
Just curious that you are downloading Collection version 6 data whereas the latest version is 6.1. See:
https://ladsweb.modaps.eosdis.nasa.gov (/alerts-and-issues/90903)
Regards,
LAADS User Services
To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.
LAADS User Services
To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.
-
- Posts: 3
- Joined: Fri Aug 26, 2022 7:43 pm America/New_York
Re: 504 gateway timeout downloading MODIS l1b data
@LaadsUserServices
Thanks, I will try that WGET command and update here if it works. I used the earthdata search tool to select this data, so I guess earthdata does not point to the most updated version. Is it possible that some of the files are just missing? The download seems to 504 timeout error on the same files every time.
Thanks, I will try that WGET command and update here if it works. I used the earthdata search tool to select this data, so I guess earthdata does not point to the most updated version. Is it possible that some of the files are just missing? The download seems to 504 timeout error on the same files every time.
-
- User Services
- Posts: 285
- Joined: Mon Sep 30, 2019 8:33 am America/New_York
- Has thanked: 3 times
Re: 504 gateway timeout downloading MODIS l1b data
Both Collections are available and are listed in earthdata search tool. It is user's choice which one is selected. All files are available except where there is a data outage.
Regards,
LAADS User Services
To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.
LAADS User Services
To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.
-
- Posts: 3
- Joined: Fri Aug 26, 2022 7:43 pm America/New_York
Re: 504 gateway timeout downloading MODIS l1b data
Okay I finally found the solution I needed, posting here in case others find it useful. I ended up using the Wget command which is given from the LAADS DAAC (which stores the data but only allows queries up to 2000 files) in combination with filenames given from an Earthdata query which doesn't have a size limit. This looked like this:
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 -i file_urls.txt --header "Authorization: Bearer PERSONAL_TOKEN_login_to_LAADS_to_get_this" -P .
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 -i file_urls.txt --header "Authorization: Bearer PERSONAL_TOKEN_login_to_LAADS_to_get_this" -P .