Unable to batch download with wget or curl

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
mkwitte
Posts: 2
Joined: Mon Sep 19, 2022 3:52 pm America/New_York
Answers: 0

Unable to batch download with wget or curl

by mkwitte » Mon Sep 19, 2022 4:26 pm America/New_York

Hello DAAC community,
I need to batch download MYD06_L2 and MYD03 files over a specific geographic region. I have done this many times in the past, but it's been about a year since my last attempt. My bearer token is currently valid. I am unable to get *any* scripts (either my own or the templates provided by LAADS DAAC tutorials) to complete a download.

Using the LAADS DAAC tutorial examples:
curl -v -H 'Authorization: Bearer $TOKEN' 'https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2011/121/MYD03.A2011121.2100.061.2018029063924.hdf' > ./MYD03.A2011121.2100.061.2018029063924.hdf
or
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2011/121/MYD03.A2011121.2100.061.2018029063924.hdf" --header "Authorization: Bearer $TOKEN" -P ./

I either get "504 Gateway Time-out" or "401 Username/Password Authentication failed" errors (only timeouts with curl, both errors with wget - first a 504 then 401). It appears that wget is redirected to an authentication link (https://ladsweb.modaps.eosdis.nasa.gov/oauth/login?redirect=...), so maybe there's an added authentication step that I'm missing? Any help here would be greatly appreciated...

Thanks,
Mikael

Tags:

mkwitte
Posts: 2
Joined: Mon Sep 19, 2022 3:52 pm America/New_York
Answers: 0

Re: Unable to batch download with wget or curl

by mkwitte » Mon Sep 19, 2022 9:41 pm America/New_York

Solved my problem. I didn't realize I had to have a separate token for LAADS, which can be obtained from https://ladsweb.modaps.eosdis.nasa.gov/profiles/#tokens for future reference.

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Unable to batch download with wget or curl

by LAADS_UserServices_M » Tue Sep 20, 2022 8:16 am America/New_York

Dear Mikael,
If you are using a valid token(generated from ladsweb) you don't need any other authentication to download data. Your wget and curl commands seems correct but probably you tried to download data at time when LAADS servers were very busy. For last several days we seem to get a spike every day starting at noon -- like someone’s big download script is starting and runs for an hour or two. The issue seems to be that the file server is slow. So, we are advising users that they should write their scripts to wait a bit then retry when they get errors. We cannot guarantee anything at this point.
In case you want to see if you are using the correct token then follow these steps:
1. Go to data folder:
For example, https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2011/121/
2. "Login" at the top of the page.
3. Click on "See wget Download Command" tab, you will see complete wget command with your token already included.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

hasan.awad
Posts: 1
Joined: Wed Dec 21, 2022 3:35 am America/New_York
Answers: 0

Re: Unable to batch download with wget or curl

by hasan.awad » Wed Dec 21, 2022 3:59 am America/New_York

I'm using Windows 10 and wget 1.21.3 64bit to get data from the LAAD DAAC Data Collection MYD021KM.

The data is subsetted in the Middle East, and I am trying to download only four tiles covering Iraq, which I mosaiced, converted to Gtiff format, then reprojected to Geographic. I submitted the order and received an email with a link to the data and a wget script; I followed the instructions in the email and added a new token.

The issue is that wget cannot access the files because it receives a 404 ERROR (.tif file not available for download), despite the fact that wget can reach your server since it downloaded the README GLDAS2.pdf file.

Please keep in mind that I attempted to download them manually by following the instructions, but it did not work and I had the same issue. Finally, I resubmitted the order, but I ran into the same issue, which I assume is due to the fact that the data I downloaded there is not geographically subset and is not in Gtiff format.
I've included a cmd window to give you a better idea of the situation.

Thank you for your consideration. I eagerly await your response!
Attachments
LAAD DAAC issue.PNG
LAAD DAAC issue.PNG (113.34 KiB) Not viewed yet

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Unable to batch download with wget or curl

by LAADS_UserServices_M » Wed Dec 21, 2022 11:44 am America/New_York

The data download issue resolved. Please try again and let us know if you still the problem.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

Post Reply