Bash script error for downloading HLS data

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
seongjunlee
Posts: 4
Joined: Sun May 19, 2024 7:58 pm America/New_York
Answers: 0

Bash script error for downloading HLS data

by seongjunlee » Sun May 19, 2024 8:06 pm America/New_York

Hello.

In order to download large amounts of HLS data at once, I was downloading the .sh file provided by NASA Earthdata and downloading it with the command "bash [filename.sh ]".
But it was downloaded well,


curl: (22) The requested URL returned error: 403

Unable to Retrieve Data

Command failed with error. Please retrieve the data manually.

https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T52SDD.2015334T022042.v2.0/HLS.S30.T52SDD.2015334T022042.v2.0.B07.tif


Like this, I'm having trouble getting errors while downloading. I want to download it at once without any errors, is there any fix?

Filters:

LP DAAC - dgolon
User Services
User Services
Posts: 338
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 24 times
Been thanked: 4 times
Contact:

Re: Bash script error for downloading HLS data

by LP DAAC - dgolon » Mon May 20, 2024 10:24 am America/New_York

Hi @seongjunlee We have had a couple of users reporting having issues with the bash script. We recommend using one of these guides instead https://github.com/nasa/LPDAAC-Data-Resources/tree/main/guides as a replacement for the bash script. Thanks --Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

seongjunlee
Posts: 4
Joined: Sun May 19, 2024 7:58 pm America/New_York
Answers: 0

Re: Bash script error for downloading HLS data

by seongjunlee » Mon May 20, 2024 10:44 pm America/New_York

Thanks for requset! And I have one question about .netrc file.

If my password include special characters such as "!" or "@",

Do I need to use URL encording when I type my password in .netrc file or just type my original password?

seongjunlee
Posts: 4
Joined: Sun May 19, 2024 7:58 pm America/New_York
Answers: 0

Re: Bash script error for downloading HLS data

by seongjunlee » Fri May 24, 2024 1:02 am America/New_York

Thank you!
I'm sorry to bother you, but I have one more question.
I'm downloading HLS data using curl through parallel (curl option "-P"), but all files size are only 27kb like below:

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 27 0 27 0 0 14 0 --:--:-- 0:00:01 --:--:-- 43
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 27 0 27 0 0 16 0 --:--:-- 0:00:01 --:--:-- 71
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 27 0 27 0 0 16 0 --:--:-- 0:00:01 --:--:-- 0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 27 0 27 0 0 16 0 --:--:-- 0:00:01 --:--:-- 44

And command I used is below:
sudo xargs -P 10 -n 1 curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n < L30_2016_2019.txt

I was wondering if the -P option was a problem, so I tried removing it, but the same problem occurred.

How can I solve it?

LP DAAC - dgolon
User Services
User Services
Posts: 338
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 24 times
Been thanked: 4 times
Contact:

Re: Bash script error for downloading HLS data

by LP DAAC - dgolon » Fri May 24, 2024 11:05 am America/New_York

Hi @seongjunlee I'll have our developers take a look. Please note, we have a holiday coming up so responses will be delayed until mid-next week. Thanks --Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

LP DAAC - dgolon
User Services
User Services
Posts: 338
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 24 times
Been thanked: 4 times
Contact:

Re: Bash script error for downloading HLS data

by LP DAAC - dgolon » Fri May 31, 2024 4:31 pm America/New_York

Hi @seongjunlee Our developers were not able to replicate what you are seeing, the -p works for them. If you would like, they would like to schedule a time to have a chat over a Teams call to discuss what you are seeing. If you are interested, please email lpdaac@usgs.gov and we can set up a time to chat. Thanks --Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

seongjunlee
Posts: 4
Joined: Sun May 19, 2024 7:58 pm America/New_York
Answers: 0

Re: Bash script error for downloading HLS data

by seongjunlee » Tue Jun 04, 2024 2:01 am America/New_York

Thank, but I solved the problem!

It is just a permission problem for my folder path.

So I used "sudo chmod-R u+w /my/path/folder" to grant write permission.

Then I tried the "sudo xargs -P 100~" command again, and it worked well!

shekoofeh_haghdoost
Posts: 1
Joined: Thu Jan 04, 2024 9:48 am America/New_York
Answers: 0

Re: Bash script error for downloading HLS data

by shekoofeh_haghdoost » Thu Jun 06, 2024 10:55 am America/New_York

I have the same problem, I can not download using the bash file in the website, and also when I am using "wget" to download data the size of the downloaded data is 10 KB while when I'm downloading manually the size of the data is 45608 KB. Please help me if possible

LP DAAC - dgolon
User Services
User Services
Posts: 338
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 24 times
Been thanked: 4 times
Contact:

Re: Bash script error for downloading HLS data

by LP DAAC - dgolon » Thu Jun 13, 2024 3:26 pm America/New_York

Hi @shekoofeh_haghdoost The smaller size could be related to data. Currently HLS data is delayed due to upstream VIIRS data issues but we are expecting more HLS data later this week. If you are looking for current scenes that could be the cause of the amount of data but if not, let us know your time period of interest and we can look into it. Thanks -- Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

LP DAAC - dgolon
User Services
User Services
Posts: 338
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 24 times
Been thanked: 4 times
Contact:

Re: Bash script error for downloading HLS data

by LP DAAC - dgolon » Fri Jun 14, 2024 4:47 pm America/New_York

Hi @shekoofeh_haghdoost Could you please email lpdaac@usgs.gov if you are still having issues? We'll be able to resolve them over email or might schedule a time to have a teams call with you. Thanks - Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

Post Reply