Download a File with cURL or Wget

Please enter here to ask a question about any NASA Science related topics!
Post Reply
njester
Posts: 13
Joined: Sat Mar 06, 2021 9:03 am America/New_York

Download a File with cURL or Wget

by njester » Mon May 24, 2021 11:35 am America/New_York

The following allows you to download data from https://asdc.larc.nasa.gov/data/ using linux, a mac, or Windows with Cygin.

1) If you haven't before, create an authentication cookie that can be used with to access files behind the EarthData login page.

Code: Select all

USERNAME=<your earthdata username>
PASSWORD=<your earthdata password>
cd ~ 
touch .netrc 
echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc 
chmod 0600 .netrc 
touch .urs_cookies 
2) You can download data with the following commands:

Code: Select all

URL=<your url here>
curl -b ~/.urs_cookies -c ~/.urs_cookies --remote-name --location  --netrc $URL
or

Code: Select all

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition $URL
You can learn more about these command line tools here:
https://www.gnu.org/software/wget/
https://curl.se/docs/
https://wiki.earthdata.nasa.gov/display ... L+And+Wget

Tags:

danpeterson
Posts: 1
Joined: Sat Feb 19, 2022 2:10 pm America/New_York

Re: Download a File with cURL or Wget

by danpeterson » Sat Feb 19, 2022 2:16 pm America/New_York

Filename Is Too Long Sometimes you get an error when trying to move, delete, or copy directories where the character count for the file path is more than 260.

Note that the words directory and folder are interchangeable. We’ll use ‘directory’ going forward. The following PowerShell cmdlets can also be used on files.

That file path is 280 characters long. So we cannot copy the directory out of there to somewhere else with the normal copy-paste method. We get the Destination Path Too Long error.

njester
Posts: 13
Joined: Sat Mar 06, 2021 9:03 am America/New_York

Re: Download a File with cURL or Wget

by njester » Tue Mar 01, 2022 11:25 am America/New_York

danpeterson wrote: Sat Feb 19, 2022 2:16 pm America/New_York Filename Is Too Long Sometimes you get an error when trying to move, delete, or copy directories where the character count for the file path is more than 260.

Note that the words directory and folder are interchangeable. We’ll use ‘directory’ going forward. The following PowerShell cmdlets can also be used on files.

That file path is 280 characters long. So we cannot copy the directory out of there to somewhere else with the normal copy-paste method. We get the Destination Path Too Long error.
Could you provide the names or links of some of the files that are too long?

Post Reply