Page 1 of 1

Wget download error

Posted: Mon Aug 26, 2024 10:48 pm America/New_York
by zcwangyu
Hi, I am trying to use wget to download the MOD35_L2 data, and the command is: wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MOD35_L2/2021//" --header "Authorization: Bearer INSERT_EDL_TOKEN_HERE" -P . But it failed with the tips as following:
--2024-08-27 10:44:00-- https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MOD35_L2/2021//
Resolving ladsweb.modaps.eosdis.nasa.gov (ladsweb.modaps.eosdis.nasa.gov)... 198.118.194.40
Connecting to ladsweb.modaps.eosdis.nasa.gov (ladsweb.modaps.eosdis.nasa.gov)|198.118.194.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 246662 (241K) [text/html]
Saving to: './MOD35_L2/2021/index.html'

MOD35_L2/2021/index 100%[===================>] 240.88K 247KB/s in 1.0s

Last-modified header missing -- time-stamps turned off.
2024-08-27 10:44:04 (247 KB/s) - './MOD35_L2/2021/index.html' saved [246662/246662]

FINISHED --2024-08-27 10:44:04--
Total wall clock time: 3.5s
Downloaded: 1 files, 241K in 1.0s (247 KB/s)

I tried the command: wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MOD35_L2/2021/" --header "Authorization: Bearer INSERT_EDL_TOKEN_HERE" -P . It also failed.
--2024-08-27 10:33:34-- https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MOD35_L2/2021/
Resolving ladsweb.modaps.eosdis.nasa.gov (ladsweb.modaps.eosdis.nasa.gov)... 198.118.194.40
Connecting to ladsweb.modaps.eosdis.nasa.gov (ladsweb.modaps.eosdis.nasa.gov)|198.118.194.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 246662 (241K) [text/html]
Saving to: './MOD35_L2/2021/index.html'

MOD35_L2/2021/index 100%[===================>] 240.88K 339KB/s in 0.7s

Last-modified header missing -- time-stamps turned off.
2024-08-27 10:33:37 (339 KB/s) - './MOD35_L2/2021/index.html' saved [246662/246662]

--2024-08-27 10:33:37-- https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MOD35_L2/2021/001/
Reusing existing connection to ladsweb.modaps.eosdis.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 415380 (406K) [text/html]
Saving to: './MOD35_L2/2021/001/index.html'

MOD35_L2/2021/001/i 100%[===================>] 405.64K 421KB/s in 1.0s

Last-modified header missing -- time-stamps turned off.
2024-08-27 10:33:40 (421 KB/s) - './MOD35_L2/2021/001/index.html' saved [415380/415380]

But!!! I can download the README text file, and download data using python script. I am trying to use wget command for downloading data. Please help me! Please Please~

Re: Wget download error

Posted: Tue Aug 27, 2024 8:35 am America/New_York
by LAADS-EDL_UserServices_M
There is a typo in the data link in the wget command on our website. In the wget command on the page it shows two forward slashes "//" at the end of the URL where there should be only one, see:
https://ladsweb.modaps.eosdis.nasa.gov/alerts-and-issues/148498
The command will work if you remove the extra slash like:
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MOD35_L2/2021/" --header "Authorization: Bearer INSERT_EDL_TOKEN_HERE" -P .