Download ASDC Files with Wget

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
joeyhotz
Posts: 4
Joined: Tue Aug 30, 2022 11:58 am America/New_York
Answers: 0

Re: Download ASDC Files with Wget

by joeyhotz » Thu Sep 01, 2022 12:31 pm America/New_York

@njester I tried the code you sent above using the TOKEN and URL variables and it worked perfectly after a bit of tweaking! Thanks for your help!

Filters:

vescovinit
Posts: 1
Joined: Fri Oct 21, 2022 4:18 am America/New_York
Answers: 0

Re: Download ASDC Files with Wget

by vescovinit » Fri Oct 21, 2022 4:52 am America/New_York

Hello,

Is there a way to download all the files on https://asdc.larc.nasa.gov/data/CALIPSO/LID_L3_Tropospheric_APro_AllSky-Standard-V4-20/ but exclude the files "D.hdf" and so only download the "N.hdf" ?

thank you,
Thomas V.

njester
Posts: 20
Joined: Sat Mar 06, 2021 9:03 am America/New_York
Answers: 0
Been thanked: 3 times

Re: Download ASDC Files with Wget

by njester » Fri Oct 21, 2022 9:42 am America/New_York

@vescovinit

Sure, you can use the "--accept" or "--reject" arguments to filter out the data you want. To use "--accept", just add the "--accept *N.hdf" argument to the command. Alternatively, you could use "--reject" by adding "--reject *D.hdf" to the command. You can read more about these commands here: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html#:~:text=The%20argument%20to%20%27%20%2D%2Daccept,%5B0%2D9%5D*%20%27.

maria.stoica
Posts: 6
Joined: Wed Mar 23, 2022 3:27 pm America/New_York
Answers: 0

Re: Download ASDC Files with Wget

by maria.stoica » Thu Jan 05, 2023 2:19 pm America/New_York

I am attempting to write a script that is scheduled to run periodically to download MOPITT data (https://asdc.larc.nasa.gov/data/MOPITT/MOP03JM.009/) as it becomes available. To my understanding, the token method requires updating the token every two months. Is there an alternate method for direct download of data where a permanent access key or password can be used instead? Thank you.

njester
Posts: 20
Joined: Sat Mar 06, 2021 9:03 am America/New_York
Answers: 0
Been thanked: 3 times

Re: Download ASDC Files with Wget

by njester » Tue Jan 10, 2023 11:49 am America/New_York

At @maria.stoica,

I don't know of a way to request a longer life token, it may be worth asking Earthdata using the Feedback button in the top-right. Another option would be to have your script use the API to check generate a new token when your current one has expired, you can find an example on this page:
https://wiki.earthdata.nasa.gov/display/EL/How+to+Generate+a+User+Token
I hope that helps.

maria.stoica
Posts: 6
Joined: Wed Mar 23, 2022 3:27 pm America/New_York
Answers: 0

Re: Download ASDC Files with Wget

by maria.stoica » Tue Jan 17, 2023 5:28 pm America/New_York

Great suggestion, thank you.

maria.stoica
Posts: 6
Joined: Wed Mar 23, 2022 3:27 pm America/New_York
Answers: 0

Re: Download ASDC Files with Wget

by maria.stoica » Fri Jun 02, 2023 5:29 pm America/New_York

Hello. I used to be able to use the wget code provided in the first post of this thread to retrieve the html content from https://asdc.larc.nasa.gov/data/MOPITT/MOP03JM.009/. I am now getting the following error: "Read error at byte 12678 (Success).Retrying." I get a similar error for other sibling directories. I do not get this error for the parent directory https://asdc.larc.nasa.gov/data/MOPITT/ or the child directories. Using Python's requests.get() yields requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)". Any help would be appreciated. Thank you.

ASDC - David W.
User Services
User Services
Posts: 65
Joined: Tue Jun 08, 2021 11:29 pm America/New_York
Answers: 0

Re: Download ASDC Files with Wget

by ASDC - David W. » Thu Aug 10, 2023 3:00 pm America/New_York

hi @maria.stoica,

just checking up on your progress. did you get your data files or are you still having problems?

thank you,

David W.
NASA Langley ASDC DAAC Lifecycle

vb1997
Posts: 2
Joined: Thu Sep 28, 2023 8:56 am America/New_York
Answers: 0

Re: Download ASDC Files with Wget

by vb1997 » Tue Oct 10, 2023 4:47 am America/New_York

Hi, I am trying to download MISR data files from the top URL https://asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/. I have followed all the steps mentioned in the first post of this thread. I was able to download CALIPSO data using the same method but now I am getting a similar error as mentioned by @maria.stoica while downloading multiple files with the URL mentioned above.
The error is as follows:
#URL used
URL=https://asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/
#I have used the token as mentioned in the first post
# wget script
wget --header "Authorization: Bearer $TOKEN" --recursive --no-parent --reject "index.html*" --execute robots=off $URL
# Output
--2023-10-10 14:03:43-- https://asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/
Resolving asdc.larc.nasa.gov (asdc.larc.nasa.gov)... 198.119.164.126, 2001:4d0:2340:4001::7e
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/index.html.tmp’

asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/index.ht [ <=> ] 12.38K 47.6KB/s in 0.3s

2023-10-10 14:04:45 (47.6 KB/s) - Read error at byte 12678 (Success).Retrying.

# This error keeps on repeating
Any help is much appreciated. Thank you.

ASDC - cheyenne.e.land
Subject Matter Expert
Subject Matter Expert
Posts: 133
Joined: Mon Mar 22, 2021 3:55 pm America/New_York
Answers: 1
Has thanked: 1 time
Been thanked: 9 times

Re: Download ASDC Files with Wget

by ASDC - cheyenne.e.land » Fri Oct 13, 2023 12:28 pm America/New_York

Hello,

We will take a look at this, thank you.

Regards

Post Reply