Earthdata Forum

To the team,

I am working on auto-downloading data from https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/. There is an error while using Wget2(https://manpages.ubuntu.com/manpages/impish/man1/wget2.1.html) to download the whole directory. Here is my code and ERROR message

wget2 --max-threads 10 --http-user=yufengho --ask-password --mirror --timestamping --continue --no-parent --no-host-directories --reject="html,tmp,jpg,csv" --robots=off --accept-regex=".h5" --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies -P /mnt/gaia/raw/GEDI/ https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/
Password:
[0] Checking 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/' ...
HTTP ERROR response 401 Unauthorized [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/]

I also tried wget for single file and it works. Code below:

wget --user yufengho --ask-password https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.13/ATL03_20181013235645_02340114_005_01.h5
Password:
....
....
ATL03_20181013235645_02340114_005_01.h5 100%[=================================================================================================================================================================================================>] 390,71M 19,4MB/s in 21s

My assumption is wget2 and wget going to different path and wget's has been fixed but wget2 did not. I have checked my username and password are correct. I am grateful if you can resolve the download issue for wget2.

Sincerely,
Yu-Feng

Dear Yu-Feng,

We recommend you try going to your EDL profile and generating a bearer token:
urs.earthdata.nasa.gov -> "Generate Token"
Then, add that to your wget2 command as: and remove the parameters.

Let us know if that works for you.

Kind regards,
Mikala

Thanks for replying, Mikala

I have not succeeded to generate token because I tried to connect to https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/ however it shows me 'n5eil01u.ecs.nsidc.org refused to connect.' It happens to my different IP.

Sincerely,
Yu-Feng

Dear Yu-Feng,

Apologies, I don't fully understand your last message. Our operations staff believe you will have better luck with your "refuse to connect" error, if you attempt your wget2 access using a bearer token. You can generate a bearer token from your Earthdata Login profile at urs.earthdata.nasa.gov. See the "Generate Token" link at the top. Then replace your user and password commands with in your wget2 command.

Would you mind giving that a try and letting me know if you are successful?

Thank you!
Mikala

P.S. We did experience a temporary outage on n5eil01u last night and earlier today that may have contributed to your "refused to connect" error early this morning.

Dear Mikala,

I am still testing wget2 on https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/ with token. I can access to the subdirectory but still have some difficulty to download the file.

Unfortunately, I did not do it properly so I got blocked of my IP. Could you please check and reopen for me? I would not send enormous requests during a short term.

Sincerely,
Yu-Feng

Hello again,

Have you tried a test with wget recently? The operations team hasn't made any changes to block your IP address. They are wondering if it's something with wget2.

Let me know if you can successfully use wget for a test granule.

Mikala

Dear Mikala,

I keep trying different configuration in wget & wget2. Although I did not succeed to download, I found a common pattern.

1. It's fine to download a single file using wget https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/ATL03_20181014000347_02350101_005_01.h5

2. If I aim to download the whole directory and it fails.

In wget: wget https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/

--2023-04-24 10:26:59-- https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/
Resolving n5eil01u.ecs.nsidc.org (n5eil01u.ecs.nsidc.org)... 128.138.97.102
Connecting to n5eil01u.ecs.nsidc.org (n5eil01u.ecs.nsidc.org)|128.138.97.102|:443... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

In wget2: wget2 --max-threads 1 --mirror --timestamping --continue --no-parent --filter-urls --no-host-directories --accept='*.h5' --reject='*.jpg,xml,json,csv' --load-cookies /root/.urs_cookies --robots=off --save-cookies /root/.urs_cookies --auth-no-challenge=on --keep-session-cookies https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/

[0] Checking 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/' ...
HTTP response 200 OK [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/]
[0] Downloading 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/' ...
Failed to read 102400 bytes (104)

I do not put my token or anything in this example in order not to expose my person information, but I did login successfully.

Thanks for your support and I appreciate very much.

Sincerely yours,
Yu-Feng

Dear Yu-Feng,

Does it work to add -r (for downloading recursively) to either the wget or wget2 commands?

Mikala