downloading CALIPSO data from one directory

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
pciren
Posts: 2
Joined: Thu Mar 04, 2021 4:41 pm America/New_York
Answers: 0

downloading CALIPSO data from one directory

by pciren » Thu Mar 04, 2021 4:48 pm America/New_York

I am using wget to get all file under a directory, but it returned nothing. The script i used as follows:
============================================================
URL=https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
USERNAME=myuserid
PASSWORD=mypassword

cd ~
touch .netrc
echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc
chmod 0600 .netrc
touch .urs_cookies
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition --recursive --wait=60 --no-parent --reject "index.html*" --execute robots=off $URL
~


==============================================================================


------------------------------------------------------------------------
the following is the returned message on screen.
------------------------------------------------------------------------------------------------------------------------------
--2021-03-04 21:34:31-- https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
Resolving asdc.larc.nasa.gov (asdc.larc.nasa.gov)... 198.119.164.126, 2001:4d0:2340:4001::7e
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?splash=false&client_id=iQGRa5KtDl_e-fgYqB5x5Q&response_type=code&redirect_uri=https%3A%2F%2Fasdc.larc.nasa.gov%2Fdata%2Furs&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8 [following]
--2021-03-04 21:35:31-- https://urs.earthdata.nasa.gov/oauth/authorize?splash=false&client_id=iQGRa5KtDl_e-fgYqB5x5Q&response_type=code&redirect_uri=https%3A%2F%2Fasdc.larc.nasa.gov%2Fdata%2Furs&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://asdc.larc.nasa.gov/data/urs?code=efdd497faef09abeba1758558518ea32a9b6a47a2a54b8a88c9e5d9b12391ebf&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8 [following]
--2021-03-04 21:36:31-- https://asdc.larc.nasa.gov/data/urs?code=efdd497faef09abeba1758558518ea32a9b6a47a2a54b8a88c9e5d9b12391ebf&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/ [following]
--2021-03-04 21:37:31-- http://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/ [following]
--2021-03-04 21:38:31-- https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/index.html’

[ <=> ] 384,362 754KB/s in 0.5s

2021-03-04 21:38:32 (754 KB/s) - ‘asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/index.html’ saved [384362]

Removing asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/index.html since it should be rejected.

FINISHED --2021-03-04 21:38:32--
Total wall clock time: 4m 1s
Downloaded: 1 files, 375K in 0.5s (754 KB/s)

Tags:

ASDC - rbeck
Posts: 17
Joined: Tue Nov 19, 2019 11:41 am America/New_York
Answers: 0

Re: downloading CALIPSO data from one directory

by ASDC - rbeck » Fri Mar 05, 2021 11:31 am America/New_York

Dear Pubu Ciren,

After reaching out to our development team I received the following guidance:
Below I’ve a script that worked for me. If that does not work, could you send me a copy of the file that it downloads?
Additionally, please verify that your earthdata login works by logging into the following site: Earthdata Login (nasa.gov)

$ URL=https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Prov-V3-40/2020/05/CAL_LID_L2_05kmAPro-Prov-V3-40.2020-05-01T01-49-51ZD.hdf
$ USERNAME=njester
$ PASSWORD=<my password>
$ cd ~
$ touch .netrc
$ echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc
$ chmod 0600 .netrc
$ touch .urs_cookies

Then one of the two methods:
$ wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies $URL
or
$ curl -b ~/.urs_cookies -c ~/.urs_cookies --remote-name --location --netrc $URL

If you continue to have difficulties, please let us know.

njester
Posts: 20
Joined: Sat Mar 06, 2021 9:03 am America/New_York
Answers: 0
Been thanked: 3 times

Re: downloading CALIPSO data from one directory

by njester » Sat Mar 06, 2021 9:04 am America/New_York

Also, it looks like the link to earthdata login didn't make it into the above post, but since you were able to log into the forum, your Earthdata login should be fine.

pciren
Posts: 2
Joined: Thu Mar 04, 2021 4:41 pm America/New_York
Answers: 0

Re: downloading CALIPSO data from one directory

by pciren » Sat Mar 06, 2021 9:53 pm America/New_York

Thanks! it works for one file, but if I would like to download all file under one directory, what should do?

I used to be able to use option "-A hdf" in wget, but it did not work any more..

Do you have any suggestion?


thanks!



-pubu

Post Reply