Wget Script to Use For Direct Data Dowload Data

Please enter here to ask a question about any NASA Science related topics!
Post Reply
Posts: 268
Joined: Thu Jul 11, 2019 4:32 pm America/New_York
Answers: 2

Wget Script to Use For Direct Data Dowload Data

by EarthdataForumContributor » Wed Jul 22, 2020 12:06 pm America/New_York

I was interested in downloading CAPLIPSO expedited data from the ASDC’s direct data download. There’s some useful information on how to use scripts to download the data, but when I tried using the wget script in the example, I’m only able to get a specific file. I was wondering if it was possible to download the entire contents of a directory, rather than individual files?


ASDC - mcook
User Services
User Services
Posts: 71
Joined: Wed Sep 25, 2019 10:53 am America/New_York

Re: Wget Script to Use For Direct Data Dowload Data

by ASDC - mcook » Wed Jul 22, 2020 12:21 pm America/New_York

Here is an example of a Wget script for downloading the entire directory.
Just substitute your product data set in place of the example (CALIPSO data set) below.

URL= https://asdc.larc.nasa.gov/data/CALIPSO/LID_L1_Exp-Prov-V3-40/
USERNAME=<your Earthdata login username>
PASSWORD=<your Earthdata login password>
cd ~
touch .netrc
echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc
chmod 0600 .netrc
touch .urs_cookies
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition --recursive --wait=60 --no-parent --reject "index.html*" --execute robots=off $URL

Post Reply