Download ASDC Files with Wget

[joeyhotz]

@njester I tried the code you sent above using the TOKEN and URL variables and it worked perfectly after a bit of tweaking! Thanks for your help!

[vescovinit]

Hello,

Is there a way to download all the files on https://asdc.larc.nasa.gov/data/CALIPSO/LID_L3_Tropospheric_APro_AllSky-Standard-V4-20/ but exclude the files "D.hdf" and so only download the "N.hdf" ?

thank you,
Thomas V.

[njester]

@vescovinit

Sure, you can use the "--accept" or "--reject" arguments to filter out the data you want. To use "--accept", just add the "--accept *N.hdf" argument to the command. Alternatively, you could use "--reject" by adding "--reject *D.hdf" to the command. You can read more about these commands here: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html#:~:text=The%20argument%20to%20%27%20%2D%2Daccept,%5B0%2D9%5D*%20%27.

[maria.stoica]

I am attempting to write a script that is scheduled to run periodically to download MOPITT data (https://asdc.larc.nasa.gov/data/MOPITT/MOP03JM.009/) as it becomes available. To my understanding, the token method requires updating the token every two months. Is there an alternate method for direct download of data where a permanent access key or password can be used instead? Thank you.

[njester]

At @maria.stoica,

I don't know of a way to request a longer life token, it may be worth asking Earthdata using the Feedback button in the top-right. Another option would be to have your script use the API to check generate a new token when your current one has expired, you can find an example on this page:
https://wiki.earthdata.nasa.gov/display/EL/How+to+Generate+a+User+Token
I hope that helps.

[maria.stoica]

Great suggestion, thank you.

[maria.stoica]

Hello. I used to be able to use the wget code provided in the first post of this thread to retrieve the html content from https://asdc.larc.nasa.gov/data/MOPITT/MOP03JM.009/. I am now getting the following error: "Read error at byte 12678 (Success).Retrying." I get a similar error for other sibling directories. I do not get this error for the parent directory https://asdc.larc.nasa.gov/data/MOPITT/ or the child directories. Using Python's requests.get() yields requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)". Any help would be appreciated. Thank you.

[ASDC - David W.]

hi @maria.stoica,

just checking up on your progress. did you get your data files or are you still having problems?

thank you,

David W.
NASA Langley ASDC DAAC Lifecycle

[vb1997]

Hi, I am trying to download MISR data files from the top URL https://asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/. I have followed all the steps mentioned in the first post of this thread. I was able to download CALIPSO data using the same method but now I am getting a similar error as mentioned by @maria.stoica while downloading multiple files with the URL mentioned above.
The error is as follows:
#URL used
URL=https://asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/
#I have used the token as mentioned in the first post
# wget script
wget --header "Authorization: Bearer $TOKEN" --recursive --no-parent --reject "index.html*" --execute robots=off $URL
# Output
--2023-10-10 14:03:43-- https://asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/
Resolving asdc.larc.nasa.gov (asdc.larc.nasa.gov)... 198.119.164.126, 2001:4d0:2340:4001::7e
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/index.html.tmp’

asdc.larc.nasa.gov/data/MISR/MIL3MAEN.004/index.ht [ <=> ] 12.38K 47.6KB/s in 0.3s

2023-10-10 14:04:45 (47.6 KB/s) - Read error at byte 12678 (Success).Retrying.

# This error keeps on repeating
Any help is much appreciated. Thank you.

[ASDC - cheyenne.e.land]

Hello,

We will take a look at this, thank you.

Regards