multi-folder download using wget download command

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
peppe
Posts: 2
Joined: Fri May 27, 2022 9:01 am America/New_York
Answers: 0

multi-folder download using wget download command

by peppe » Fri May 27, 2022 9:21 am America/New_York

Hi all, I'm trying to realize multi-file (multi-folder) download using wget download command for specific days from the folder https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2020/.
The command works properly when downloading multi-file for only one day.
On the other hand, I am not able to download data for determined multiple days (e.g., using list option for wget).
In particular, after completing the multi-file download of a specific day, lets' say day 091, the system starts automatically the downloads from day 001, after creating all the 365 daily folders.
Any suggestion to avoid this?

I would like wget to enter a list of URL (e.g., using wget -i url.txt in the command line) and let's say download files into folder 092 and 093. But wget always starts to download day 001 files after completing the day 091.

Many thanks for your kind help,
Peppe

Tags:

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: multi-folder download using wget download command

by LAADS_UserServices_M » Tue May 31, 2022 6:53 am America/New_York

Dear Peppe:
If you are using the URL in your wget command the way you sent (https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2020/) it is going to download all days of the year starting from day 001. However, if you are using a different URL, send your complete wget command (without your token) for us to investigate.

Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

peppe
Posts: 2
Joined: Fri May 27, 2022 9:01 am America/New_York
Answers: 0

Re: multi-folder download using wget download command

by peppe » Tue May 31, 2022 10:19 am America/New_York

Many thanks for your support.
The wget command I'm using is:

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD21/2012/061" --header "Authorization: OMISSIS" -P F:\


Alternatively, I tried using the command wget -i list.txt with the same results (i.e., downloading day 061 and after that all days since 001.

In the list.txt I placed all the URL of my interest, such as:
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD21/2012/061
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD21/2012/091
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD21/2012/121
... etc...

Hope you can help me.
Best,
Peppe

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: multi-folder download using wget download command

by LAADS_UserServices_M » Fri Jun 03, 2022 4:31 pm America/New_York

Dear Peppe:
Thanks very much for sending the details. We investigate and will get back to you ASAP.

Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

Post Reply