401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

[yufengho]

To the team,

I am working on auto-downloading data from https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/. There is an error while using Wget2(https://manpages.ubuntu.com/manpages/impish/man1/wget2.1.html) to download the whole directory. Here is my code and ERROR message

wget2 --max-threads 10 --http-user=yufengho --ask-password --mirror --timestamping --continue --no-parent --no-host-directories --reject="html,tmp,jpg,csv" --robots=off --accept-regex=".h5" --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies -P /mnt/gaia/raw/GEDI/ https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/
Password:
[0] Checking 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/' ...
HTTP ERROR response 401 Unauthorized [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/]

I also tried wget for single file and it works. Code below:

wget --user yufengho --ask-password https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.13/ATL03_20181013235645_02340114_005_01.h5
Password:
....
....
ATL03_20181013235645_02340114_005_01.h5 100%[=================================================================================================================================================================================================>] 390,71M 19,4MB/s in 21s

My assumption is wget2 and wget going to different path and wget's has been fixed but wget2 did not. I have checked my username and password are correct. I am grateful if you can resolve the download issue for wget2.

Sincerely,
Yu-Feng

[NSIDC - mbeig]

Dear Yu-Feng,

We recommend you try going to your EDL profile and generating a bearer token:
urs.earthdata.nasa.gov -> "Generate Token"
Then, add that to your wget2 command as:

Code: Select all

wget2 --header='Authorization: Bearer TOKEN_HERE'

and remove the

Code: Select all

--http-user=yufengho --ask-password

parameters.

Let us know if that works for you.

Kind regards,
Mikala

[yufengho]

Thanks for replying, Mikala

I have not succeeded to generate token because I tried to connect to https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/ however it shows me 'n5eil01u.ecs.nsidc.org refused to connect.' It happens to my different IP.

Sincerely,
Yu-Feng

[NSIDC - mbeig]

Dear Yu-Feng,

Apologies, I don't fully understand your last message. Our operations staff believe you will have better luck with your "refuse to connect" error, if you attempt your wget2 access using a bearer token. You can generate a bearer token from your Earthdata Login profile at urs.earthdata.nasa.gov. See the "Generate Token" link at the top. Then replace your user and password commands with

Code: Select all

--header='Authorization: Bearer TOKEN_HERE'

in your wget2 command.

Would you mind giving that a try and letting me know if you are successful?

Thank you!
Mikala

[NSIDC - mbeig]

P.S. We did experience a temporary outage on n5eil01u last night and earlier today that may have contributed to your "refused to connect" error early this morning.

[yufengho]

Dear Mikala,

I am still testing wget2 on https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/ with token. I can access to the subdirectory but still have some difficulty to download the file.

Unfortunately, I did not do it properly so I got blocked of my IP. Could you please check and reopen for me? I would not send enormous requests during a short term.

Sincerely,
Yu-Feng

[NSIDC - mbeig]

Hello again,

Have you tried a test with wget recently? The operations team hasn't made any changes to block your IP address. They are wondering if it's something with wget2.

Let me know if you can successfully use wget for a test granule.

Mikala

[yufengho]

Dear Mikala,

I keep trying different configuration in wget & wget2. Although I did not succeed to download, I found a common pattern.

1. It's fine to download a single file using wget https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/ATL03_20181014000347_02350101_005_01.h5

2. If I aim to download the whole directory and it fails.

In wget: wget https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/

--2023-04-24 10:26:59-- https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/
Resolving n5eil01u.ecs.nsidc.org (n5eil01u.ecs.nsidc.org)... 128.138.97.102
Connecting to n5eil01u.ecs.nsidc.org (n5eil01u.ecs.nsidc.org)|128.138.97.102|:443... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

In wget2: wget2 --max-threads 1 --mirror --timestamping --continue --no-parent --filter-urls --no-host-directories --accept='*.h5' --reject='*.jpg,xml,json,csv' --load-cookies /root/.urs_cookies --robots=off --save-cookies /root/.urs_cookies --auth-no-challenge=on --keep-session-cookies https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/

[0] Checking 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/' ...
HTTP response 200 OK [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/]
[0] Downloading 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/' ...
Failed to read 102400 bytes (104)

I do not put my token or anything in this example in order not to expose my person information, but I did login successfully.

Thanks for your support and I appreciate very much.

Sincerely yours,
Yu-Feng

[NSIDC - mbeig]

Dear Yu-Feng,

Does it work to add -r (for downloading recursively) to either the wget or wget2 commands?

Mikala