401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
yufengho
Posts: 4
Joined: Tue Apr 18, 2023 8:48 am America/New_York
Answers: 0

401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

by yufengho » Tue Apr 18, 2023 9:03 am America/New_York

To the team,

I am working on auto-downloading data from https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/. There is an error while using Wget2(https://manpages.ubuntu.com/manpages/impish/man1/wget2.1.html) to download the whole directory. Here is my code and ERROR message

wget2 --max-threads 10 --http-user=yufengho --ask-password --mirror --timestamping --continue --no-parent --no-host-directories --reject="html,tmp,jpg,csv" --robots=off --accept-regex=".h5" --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies -P /mnt/gaia/raw/GEDI/ https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/
Password:
[0] Checking 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/' ...
HTTP ERROR response 401 Unauthorized [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/]

I also tried wget for single file and it works. Code below:

wget --user yufengho --ask-password https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.13/ATL03_20181013235645_02340114_005_01.h5
Password:
....
....
ATL03_20181013235645_02340114_005_01.h5 100%[=================================================================================================================================================================================================>] 390,71M 19,4MB/s in 21s

My assumption is wget2 and wget going to different path and wget's has been fixed but wget2 did not. I have checked my username and password are correct. I am grateful if you can resolve the download issue for wget2.

Sincerely,
Yu-Feng

Tags:

NSIDC - mbeig
User Services
User Services
Posts: 13
Joined: Tue Dec 07, 2021 11:49 am America/New_York
Answers: 0

Re: 401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

by NSIDC - mbeig » Tue Apr 18, 2023 6:20 pm America/New_York

Dear Yu-Feng,

We recommend you try going to your EDL profile and generating a bearer token:
urs.earthdata.nasa.gov -> "Generate Token"
Then, add that to your wget2 command as:

Code: Select all

wget2 --header='Authorization: Bearer TOKEN_HERE'
and remove the

Code: Select all

--http-user=yufengho --ask-password
parameters.

Let us know if that works for you.

Kind regards,
Mikala

yufengho
Posts: 4
Joined: Tue Apr 18, 2023 8:48 am America/New_York
Answers: 0

Re: 401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

by yufengho » Wed Apr 19, 2023 5:40 am America/New_York

Thanks for replying, Mikala

I have not succeeded to generate token because I tried to connect to https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/ however it shows me 'n5eil01u.ecs.nsidc.org refused to connect.' It happens to my different IP.

Sincerely,
Yu-Feng

NSIDC - mbeig
User Services
User Services
Posts: 13
Joined: Tue Dec 07, 2021 11:49 am America/New_York
Answers: 0

Re: 401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

by NSIDC - mbeig » Wed Apr 19, 2023 5:15 pm America/New_York

Dear Yu-Feng,

Apologies, I don't fully understand your last message. Our operations staff believe you will have better luck with your "refuse to connect" error, if you attempt your wget2 access using a bearer token. You can generate a bearer token from your Earthdata Login profile at urs.earthdata.nasa.gov. See the "Generate Token" link at the top. Then replace your user and password commands with

Code: Select all

--header='Authorization: Bearer TOKEN_HERE'
in your wget2 command.

Would you mind giving that a try and letting me know if you are successful?

Thank you!
Mikala

NSIDC - mbeig
User Services
User Services
Posts: 13
Joined: Tue Dec 07, 2021 11:49 am America/New_York
Answers: 0

Re: 401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

by NSIDC - mbeig » Wed Apr 19, 2023 5:17 pm America/New_York

P.S. We did experience a temporary outage on n5eil01u last night and earlier today that may have contributed to your "refused to connect" error early this morning.

yufengho
Posts: 4
Joined: Tue Apr 18, 2023 8:48 am America/New_York
Answers: 0

Re: 401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

by yufengho » Fri Apr 21, 2023 6:06 am America/New_York

Dear Mikala,

I am still testing wget2 on https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/ with token. I can access to the subdirectory but still have some difficulty to download the file.

Unfortunately, I did not do it properly so I got blocked of my IP. Could you please check and reopen for me? I would not send enormous requests during a short term.

Sincerely,
Yu-Feng

NSIDC - mbeig
User Services
User Services
Posts: 13
Joined: Tue Dec 07, 2021 11:49 am America/New_York
Answers: 0

Re: 401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

by NSIDC - mbeig » Fri Apr 21, 2023 4:38 pm America/New_York

Hello again,

Have you tried a test with wget recently? The operations team hasn't made any changes to block your IP address. They are wondering if it's something with wget2.

Let me know if you can successfully use wget for a test granule.

Mikala

yufengho
Posts: 4
Joined: Tue Apr 18, 2023 8:48 am America/New_York
Answers: 0

Re: 401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

by yufengho » Mon Apr 24, 2023 4:44 am America/New_York

Dear Mikala,

I keep trying different configuration in wget & wget2. Although I did not succeed to download, I found a common pattern.

1. It's fine to download a single file using wget https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/ATL03_20181014000347_02350101_005_01.h5

2. If I aim to download the whole directory and it fails.

In wget: wget https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/

--2023-04-24 10:26:59-- https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/
Resolving n5eil01u.ecs.nsidc.org (n5eil01u.ecs.nsidc.org)... 128.138.97.102
Connecting to n5eil01u.ecs.nsidc.org (n5eil01u.ecs.nsidc.org)|128.138.97.102|:443... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

In wget2: wget2 --max-threads 1 --mirror --timestamping --continue --no-parent --filter-urls --no-host-directories --accept='*.h5' --reject='*.jpg,xml,json,csv' --load-cookies /root/.urs_cookies --robots=off --save-cookies /root/.urs_cookies --auth-no-challenge=on --keep-session-cookies https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/

[0] Checking 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/' ...
HTTP response 200 OK [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/]
[0] Downloading 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/' ...
Failed to read 102400 bytes (104)

I do not put my token or anything in this example in order not to expose my person information, but I did login successfully.

Thanks for your support and I appreciate very much.

Sincerely yours,
Yu-Feng

NSIDC - mbeig
User Services
User Services
Posts: 13
Joined: Tue Dec 07, 2021 11:49 am America/New_York
Answers: 0

Re: 401 Unauthorized in [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/] using Wget2

by NSIDC - mbeig » Wed Apr 26, 2023 5:36 pm America/New_York

Dear Yu-Feng,

Does it work to add -r (for downloading recursively) to either the wget or wget2 commands?

Mikala

Post Reply