Download Multiple ASDC Files with Wget

[njester]

The following allows you to download data from https://asdc.larc.nasa.gov/data/ using linux, a mac, or Windows with Cygin.

1) If you haven't before, create an authentication cookie that can be used with to access files behind the EarthData login page. Please note, if you already have a .netrc file, instead of running the line that starts with echo, open the .netrc file with the editor of your choice and add "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" (replace the words starting with $ with your Earthdata login credentials)

Code: Select all

USERNAME=<your earthdata username>
PASSWORD=<your earthdata password>
cd ~ 
touch .netrc 
echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc 
chmod 0600 .netrc 
touch .urs_cookies 

2) Next, you'll need a URL for the data you want to download, to get a url. To find a data url, head here https://asdc.larc.nasa.gov/data/ and search for the data you want to download and add that URL to the first line. You can download data with the following commands:

Code: Select all

URL=<the data url goes here>
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition  --recursive --wait=60 --no-parent --reject "index.html*" --execute robots=off $URL

Recursive wget is a fairly complex and powerful with lots of useful options for things like filtering results. I've provided an explanation for the options that I chose below.

The following arguments are required to authenticate:

• --load-cookies ~/.urs_cookies
• --save-cookies ~/.urs_cookies
• --auth-no-challenge=on
• --keep-session-cookies
• --content-disposition
• --no-check-certificate

Other Arguments:

1. --recursive: To recursively download files, wget performs the following process:
   1. Downloads the requested page, this is the first level (like the root of a tree file tree)
   2. Scans the document(s) downloaded this level for links
   3. Downloads all the links found in this level, these contents are a new level
   4. Repeat steps 2 and 3 for <depth> levels defined by the --level=depth argument. If undefined, depth is set to 5.
2. wait=60: Wait the specified number of seconds (in this case, 60) between the retrievals. From the documentation at https://www.gnu.org/software/wget/manua ... index-wait
   “You should be warned that recursive downloads can overload the remote servers...consider using the
3. -w option to introduce a delay between accesses to the server. The download will take a while longer, but the server administrator will not be alarmed by your rudeness.” –Wget Documentation
4. --no-parent: ignores any links that point to a location above the current level (such as a link to the top-level site)
5. --reject "index.html*": (optional) does not download the contents of the index page, it is unlikely that users will want this page's contents, but instead will want the data in the links on the page.
6. --execute robots=off: this argument tells wget to ignore robots.txt, a file which tells automated tools where not to search.
7. --accept=”<pattern>”: this argument filters files based on a file name pattern.
8. --accept-regex=”<pattern>”: this argument filters files by URL using regex.

Wget has lots of options for, so if you need something that the script above doesn't do, check the official documentation:
https://www.gnu.org/software/wget/

[lefsky]

I am trying to adapt this script to allow me to download all of the file locations within a particular directory, specifically

https://asdc.larc.nasa.gov/data/DSCOVR/EPIC/L2_CLOUD_03

using the --spider option but it only downloads the index.html.* files rather than rejecting them. Can you assist?

[ASDC - cheyenne.e.land]

Hello,

Thank you for your question. A Subject Matter Expert has been notified and will answer your question shortly.

[njester]

I am trying to adapt this script to allow me to download all of the file locations within a particular directory, specifically

https://asdc.larc.nasa.gov/data/DSCOVR/EPIC/L2_CLOUD_03

using the --spider option but it only downloads the index.html.* files rather than rejecting them. Can you assist?

The script provided should download all the files within a directory. Did you use the first code from the first code block to create .netrc and .urs_cookies file? If not, or if the credentials you entered were wrong, it will download an html page saying that it login failed. I don't think you'll need spider for this application unless I've misunderstood your question.
-Nathan

[roux]

Note that there is a gotcha in this recipe. This line

"""
echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc
"""

will overwrite whatever you had in your ~/.netrc.

Backup often and stay safe :)

[njester]

Note that there is a gotcha in this recipe. This line

"""
echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc
"""

will overwrite whatever you had in your ~/.netrc.

Backup often and stay safe

That's a great point! I'll update the code example!

[mingquan]

njester wrote:
> [quote=lefsky post_id=8918 time=1628879738 user_id=3801]
> I am trying to adapt this script to allow me to download all of the file
> locations within a particular directory, specifically
>
> https://asdc.larc.nasa.gov/data/DSCOVR/EPIC/L2_CLOUD_03
>
> using the --spider option but it only downloads the index.html.* files
> rather than rejecting them. Can you assist?
> [/quote]
>
> The script provided should download all the files within a directory. Did
> you use the first code from the first code block to create .netrc and
> .urs_cookies file? If not, or if the credentials you entered were wrong,
> it will download an html page saying that it login failed. I don't think
> you'll need spider for this application unless I've misunderstood your
> question.
> -Nathan

The first code works well. However the issue is in the second code. How to set up my URL in the second code? I have server, redwood.ess.uci.edu. After I define: URL = redwood.ess.uci.edu, it seems like my server has no response at all when I run wget, the last line in the second code.

[njester]

I think I may have used poor wording in the example, thank you for pointing this out. I can see where "URL=<your url here>" is could be confusing, because it's not your personal URL, but instead it's the URL of the files you want to download. I'll update the wording later today. Thank you!

Here's an example of how to download code:

Lets say you wanted to download the contents of

Code: Select all

https://asdc.larc.nasa.gov/data/ACEPOL/AircraftRemoteSensing_AirHARP_Data_1/

You would change

Code: Select all

URL=<your url here>

to

Code: Select all

URL=https://asdc.larc.nasa.gov/data/ACEPOL/AircraftRemoteSensing_AirHARP_Data_1/