wget syntax to download a whole directory

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
rpalikonda
Posts: 3
Joined: Tue Apr 06, 2021 3:11 pm America/New_York
Answers: 0

wget syntax to download a whole directory

by rpalikonda » Tue Apr 06, 2021 3:16 pm America/New_York

Hi,
Using Direct Data Download, I want to use wget to get all the files and subdirectory. For e.g
I want to get all the subdirectories and files under 2002.

https://asdc.larc.nasa.gov/data/CERES/G ... _V01/2002/

If sub-folders is not possible then all the files under the directory

https://asdc.larc.nasa.gov/data/CERES/G ... 1/2002/001

Thanks
Rabi

Tags:

ASDC - joseph.f.koch
User Services
User Services
Posts: 33
Joined: Mon Nov 23, 2020 3:57 pm America/New_York
Answers: 2

Re: wget syntax to download a whole directory

by ASDC - joseph.f.koch » Wed Apr 07, 2021 12:56 pm America/New_York

Hi Rabi,

Here are some instructions for downloading ASDC data using Wget:

viewtopic.php?f=7&t=714&p=1565&hilit=Da ... 289a#p1565

You may also find the 'Scripts for Downloading Data' useful which contains instructions as well. You can find these scripts on the main webpage of our Direct Data Download site, in the upper right hand corner https://asdc.larc.nasa.gov/data/

Please let us know if you have any additional questions or need further assistance.

Thanks,
Joe Koch
NASA Langley ASDC User Services

rpalikonda
Posts: 3
Joined: Tue Apr 06, 2021 3:11 pm America/New_York
Answers: 0

Re: wget syntax to download a whole directory

by rpalikonda » Tue Apr 13, 2021 2:35 pm America/New_York

Hi Joe,

I followed the directions in the ASDC direct download link. Created the .netrc, included the line with username and passwd. When I execute the following cmd below is a html file called CER_GEO_Ed4_GOE08_NH_V01_2002.001.0045.08K.nc

that is 12731 bytes. What am I doing wrong. Thx for your help.

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies https://asdc.larc.nasa.gov/data/CERES/G ... 045.08K.nc

--2021-04-13 14:32:31-- https://asdc.larc.nasa.gov/data/CERES/G ... 045.08K.nc
Resolving asdc.larc.nasa.gov (asdc.larc.nasa.gov)... 198.119.164.126
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/au ... UuMDhLLm5j [following]
--2021-04-13 14:32:31-- https://urs.earthdata.nasa.gov/oauth/au ... UuMDhLLm5j
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: â

CER_GEO_Ed4_GOE08_NH_V01_2002.001. [ <=> ] 12.43K --.-KB/s in 0.02s

2021-04-13 14:32:31 (667 KB/s) - â saved [12731]

ASDC - joseph.f.koch
User Services
User Services
Posts: 33
Joined: Mon Nov 23, 2020 3:57 pm America/New_York
Answers: 2

Re: wget syntax to download a whole directory

by ASDC - joseph.f.koch » Thu Apr 15, 2021 11:09 am America/New_York

Hi Rabi,

Please try replacing your Wget line with the following command:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition $URL

Hope this helps, and please let us know if you run into any further issues with accessing your files.

Thanks,
Joe Koch
NASA Langley ASDC User Services

rpalikonda
Posts: 3
Joined: Tue Apr 06, 2021 3:11 pm America/New_York
Answers: 0

Re: wget syntax to download a whole directory

by rpalikonda » Thu Apr 15, 2021 11:37 am America/New_York

Joe,

Thanks!. That worked to download a single file. If I wanted to get the all the files under "001" directory what would the syntax be.I tried the '-r' option with no luck.
Rabi

aseifert
Posts: 1
Joined: Sat Apr 17, 2021 6:09 am America/New_York
Answers: 0

Re: wget syntax to download a whole directory

by aseifert » Sat Apr 17, 2021 6:12 am America/New_York

Hi,

I had similar problems with CERES data and was lucky enough to find your thread.
This works for me for a whole directory of FlashFlux SSF:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition -r -np -nc -nH --cut-dirs=4 -e robots=off https://asdc.larc.nasa.gov/data/CERES/F ... A/2021/01/

Best regards, Axel
Last edited by aseifert on Sat Apr 17, 2021 6:16 am America/New_York, edited 1 time in total.

Post Reply