Bash script error for downloading HLS data

[seongjunlee]

Hello.

In order to download large amounts of HLS data at once, I was downloading the .sh file provided by NASA Earthdata and downloading it with the command "bash [filename.sh ]".
But it was downloaded well,


curl: (22) The requested URL returned error: 403

Unable to Retrieve Data

Command failed with error. Please retrieve the data manually.

https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T52SDD.2015334T022042.v2.0/HLS.S30.T52SDD.2015334T022042.v2.0.B07.tif


Like this, I'm having trouble getting errors while downloading. I want to download it at once without any errors, is there any fix?

[LP DAAC - dgolon]

Hi @seongjunlee We have had a couple of users reporting having issues with the bash script. We recommend using one of these guides instead https://github.com/nasa/LPDAAC-Data-Resources/tree/main/guides as a replacement for the bash script. Thanks --Danielle

[seongjunlee]

Thanks for requset! And I have one question about .netrc file.

If my password include special characters such as "!" or "@",

Do I need to use URL encording when I type my password in .netrc file or just type my original password?

[seongjunlee]

Thank you!
I'm sorry to bother you, but I have one more question.
I'm downloading HLS data using curl through parallel (curl option "-P"), but all files size are only 27kb like below:

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 27 0 27 0 0 14 0 --:--:-- 0:00:01 --:--:-- 43
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 27 0 27 0 0 16 0 --:--:-- 0:00:01 --:--:-- 71
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 27 0 27 0 0 16 0 --:--:-- 0:00:01 --:--:-- 0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 27 0 27 0 0 16 0 --:--:-- 0:00:01 --:--:-- 44

And command I used is below:
sudo xargs -P 10 -n 1 curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n < L30_2016_2019.txt

I was wondering if the -P option was a problem, so I tried removing it, but the same problem occurred.

How can I solve it?

[LP DAAC - dgolon]

Hi @seongjunlee I'll have our developers take a look. Please note, we have a holiday coming up so responses will be delayed until mid-next week. Thanks --Danielle

[LP DAAC - dgolon]

Hi @seongjunlee Our developers were not able to replicate what you are seeing, the -p works for them. If you would like, they would like to schedule a time to have a chat over a Teams call to discuss what you are seeing. If you are interested, please email lpdaac@usgs.gov and we can set up a time to chat. Thanks --Danielle

[seongjunlee]

Thank, but I solved the problem!

It is just a permission problem for my folder path.

So I used "sudo chmod-R u+w /my/path/folder" to grant write permission.

Then I tried the "sudo xargs -P 100~" command again, and it worked well!

[shekoofeh_haghdoost]

I have the same problem, I can not download using the bash file in the website, and also when I am using "wget" to download data the size of the downloaded data is 10 KB while when I'm downloading manually the size of the data is 45608 KB. Please help me if possible

[LP DAAC - dgolon]

Hi @shekoofeh_haghdoost The smaller size could be related to data. Currently HLS data is delayed due to upstream VIIRS data issues but we are expecting more HLS data later this week. If you are looking for current scenes that could be the cause of the amount of data but if not, let us know your time period of interest and we can look into it. Thanks -- Danielle

[LP DAAC - dgolon]

Hi @shekoofeh_haghdoost Could you please email lpdaac@usgs.gov if you are still having issues? We'll be able to resolve them over email or might schedule a time to have a teams call with you. Thanks - Danielle