I am working on auto-downloading data from https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/. There is an error while using Wget2(https://manpages.ubuntu.com/manpages/impish/man1/wget2.1.html) to download the whole directory. Here is my code and ERROR message
wget2 --max-threads 10 --http-user=yufengho --ask-password --mirror --timestamping --continue --no-parent --no-host-directories --reject="html,tmp,jpg,csv" --robots=off --accept-regex=".h5" --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies -P /mnt/gaia/raw/GEDI/ https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/
 Checking 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/' ...
HTTP ERROR response 401 Unauthorized [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/]
I also tried wget for single file and it works. Code below:
wget --user yufengho --ask-password https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.13/ATL03_20181013235645_02340114_005_01.h5
ATL03_20181013235645_02340114_005_01.h5 100%[=================================================================================================================================================================================================>] 390,71M 19,4MB/s in 21s
My assumption is wget2 and wget going to different path and wget's has been fixed but wget2 did not. I have checked my username and password are correct. I am grateful if you can resolve the download issue for wget2.
We recommend you try going to your EDL profile and generating a bearer token:
urs.earthdata.nasa.gov -> "Generate Token"
Then, add that to your wget2 command as:
Code: Select all
wget2 --header='Authorization: Bearer TOKEN_HERE'
Code: Select all
Let us know if that works for you.
I have not succeeded to generate token because I tried to connect to https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/ however it shows me 'n5eil01u.ecs.nsidc.org refused to connect.' It happens to my different IP.
Apologies, I don't fully understand your last message. Our operations staff believe you will have better luck with your "refuse to connect" error, if you attempt your wget2 access using a bearer token. You can generate a bearer token from your Earthdata Login profile at urs.earthdata.nasa.gov. See the "Generate Token" link at the top. Then replace your user and password commands with
Code: Select all
--header='Authorization: Bearer TOKEN_HERE'
Would you mind giving that a try and letting me know if you are successful?
I am still testing wget2 on https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/ with token. I can access to the subdirectory but still have some difficulty to download the file.
Unfortunately, I did not do it properly so I got blocked of my IP. Could you please check and reopen for me? I would not send enormous requests during a short term.
Have you tried a test with wget recently? The operations team hasn't made any changes to block your IP address. They are wondering if it's something with wget2.
Let me know if you can successfully use wget for a test granule.
I keep trying different configuration in wget & wget2. Although I did not succeed to download, I found a common pattern.
1. It's fine to download a single file using wget https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/ATL03_20181014000347_02350101_005_01.h5
2. If I aim to download the whole directory and it fails.
In wget: wget https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/
--2023-04-24 10:26:59-- https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/
Resolving n5eil01u.ecs.nsidc.org (n5eil01u.ecs.nsidc.org)... 188.8.131.52
Connecting to n5eil01u.ecs.nsidc.org (n5eil01u.ecs.nsidc.org)|184.108.40.206|:443... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
In wget2: wget2 --max-threads 1 --mirror --timestamping --continue --no-parent --filter-urls --no-host-directories --accept='*.h5' --reject='*.jpg,xml,json,csv' --load-cookies /root/.urs_cookies --robots=off --save-cookies /root/.urs_cookies --auth-no-challenge=on --keep-session-cookies https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.14/
 Checking 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/' ...
HTTP response 200 OK [https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/]
 Downloading 'https://n5eil01u.ecs.nsidc.org/ATLAS/ATL03.005/2018.10.15/' ...
Failed to read 102400 bytes (104)
I do not put my token or anything in this example in order not to expose my person information, but I did login successfully.
Thanks for your support and I appreciate very much.
Does it work to add -r (for downloading recursively) to either the wget or wget2 commands?