ASTER Data Download Cygwin Unable to Retrieve Data

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
srleone9
Posts: 4
Joined: Mon Mar 08, 2021 12:34 pm America/New_York
Answers: 0

ASTER Data Download Cygwin Unable to Retrieve Data

by srleone9 » Mon Mar 08, 2021 12:52 pm America/New_York

Novice coder here, read: Windows 10 user who knows what a DOS prompt is, lol
...
I am wracking my brain trying to use the Cygwin console to download many, many thousands of files to add to a GIS (using QGIS).
I have successfully downloaded the shell file script (download.sh) and after some digging in the Cygwin user guide, figured out where to save it to take advantage of Cygwin's default file paths.
However, it seems no matter how many applications I authorize in my Earthdata profile, after entering my Earthdata login credentials, I get the same message that refers to the same URL, which is just an all-black tif file.
Here's the error message itself:
Unable to Retrieve Data
Please ensure that you have authorized the remote application by visiting the link below
https://lpdaac.earthdata.nasa.gov/lp-pr ... 80_dem.tif

I found directions on authorizing apps at https://wiki.earthdata.nasa.gov/display ... pplication
I looked at the download script itself in Notepad, and it appears that the error message is not dynamic, so methinks it's not really a useful URL...I mined the script file for letter combos in applications to authorize but, as I said, no change in the error message.
...
What is the magic remote application that must be authorized?
How do I get my data?!? HELP please :-)

Thanks, Su

Tags:

LP DAAC - jwilson
User Services
User Services
Posts: 176
Joined: Mon Sep 30, 2019 12:39 pm America/New_York
Answers: 1
Has thanked: 7 times

Re: ASTER Data Download Cygwin Unable to Retrieve Data

by LP DAAC - jwilson » Mon Mar 08, 2021 3:12 pm America/New_York

After you create your account, you will also need to “authorize” the LP DAAC Data Pool application. Once the Data Pool application has been authorized, you will be able to download data.

Please let me know if you continue to have issues.
Kind regards,
Janice

srleone9
Posts: 4
Joined: Mon Mar 08, 2021 12:34 pm America/New_York
Answers: 0

Re: ASTER Data Download Cygwin Unable to Retrieve Data

by srleone9 » Mon Mar 08, 2021 4:00 pm America/New_York

Janice, that was the first app I authorized, since it's mentioned in that old wiki article.
Hence, my absolute confusion. I ended up downloading ETOPO1, but the resolution is not as good.
Any other pointers?
Thanks,
Su

LP DAAC - jwilson
User Services
User Services
Posts: 176
Joined: Mon Sep 30, 2019 12:39 pm America/New_York
Answers: 1
Has thanked: 7 times

Re: ASTER Data Download Cygwin Unable to Retrieve Data

by LP DAAC - jwilson » Tue Mar 09, 2021 10:56 am America/New_York

Dear Su,
We searched the Earthdata Search client for the granule ASTGTMV003N36E080.zip and downloaded it with no issues. We could not duplicate your issue with the data not being available. Please see attachment.

LP DAAC does not support software and I'm not familiar with the Cygwin software. You might need to contact their support for further guidance.

I hope this helps,
Kind regards,
Janice
Attachments
ASTGTMV003_N36E080.jpg
ASTGTMV003_N36E080.jpg (46.04 KiB) Not viewed yet

srleone9
Posts: 4
Joined: Mon Mar 08, 2021 12:34 pm America/New_York
Answers: 0

Re: ASTER Data Download Cygwin Unable to Retrieve Data

by srleone9 » Tue Mar 09, 2021 11:48 am America/New_York

Downloading one at a time is not the problem, nor is that specific granule.
Cygwin is what Earthdata suggests for downloading using a script. Otherwise it would take a year. Literally.
For some reason, that granule is hard-coded into the script generated by Earthdata as the output for all "unable to retrieve" error messages.

Any ideas?

LP DAAC - jwilson
User Services
User Services
Posts: 176
Joined: Mon Sep 30, 2019 12:39 pm America/New_York
Answers: 1
Has thanked: 7 times

Re: ASTER Data Download Cygwin Unable to Retrieve Data

by LP DAAC - jwilson » Wed Mar 10, 2021 5:02 pm America/New_York

Hi Su,
You might benefit from using the AppEEARS tool.

AppEEARS landing page: https://lpdaacsvc.cr.usgs.gov/appeears/
You will need to use your Earthdata Login
Getting started instructions can be found here: https://lpdaacsvc.cr.usgs.gov/appeears/help

I'm going to forward your question to our subject matter expert for further guidance.
Thank you for your patience.
Janice

roblox_free_robux
Posts: 1
Joined: Tue Mar 09, 2021 12:59 am America/New_York
Answers: 0

Re: ASTER Data Download Cygwin Unable to Retrieve Data

by roblox_free_robux » Thu Mar 11, 2021 3:23 am America/New_York

Hi Mate,

Thank You, For Sharing The Information and Really Appreciated

LP DAAC - ckrehbiel
Subject Matter Expert
Subject Matter Expert
Posts: 59
Joined: Tue Nov 12, 2019 3:29 pm America/New_York
Answers: 0

Re: ASTER Data Download Cygwin Unable to Retrieve Data

by LP DAAC - ckrehbiel » Thu Mar 11, 2021 11:48 am America/New_York

@@srleone9 I unfortunately can't help with the cygwin issue, for that specifically, I would defer to: support@earthdata.nasa.gov

However, it looks like your use case is that you'd like to bulk download many different ASTER GDEM files directly from the LP DAAC Data Pool? There are a couple of solutions I can point you towards.
First, once you have the desired files in Earthdata Search, ex: https://search.earthdata.nasa.gov/proje ... 99669636!4!!

You will see this page:
2021-03-11_10-41-21.png
2021-03-11_10-41-21.png (34.79 KiB) Not viewed yet
Go ahead and save the list of links as a text file. Next, we have multiple different options that will allow you to bulk download these files, including:

How to bulk download from the command line using wget or curl: https://lpdaac.usgs.gov/resources/e-lea ... mand-line/

How to Access the LP DAAC Data Pool with Python: https://git.earthdata.nasa.gov/projects ... hon/browse

How to Access the LP DAAC Data Pool with R:
https://git.earthdata.nasa.gov/projects ... d_r/browse

You should be able to feed that text file (using the instructions provided for each of the options) into any of the options listed above in order to bulk download your desired files.

Let me know if you are still encountering issues after trying these options.

Thanks!
Cole

srleone9
Posts: 4
Joined: Mon Mar 08, 2021 12:34 pm America/New_York
Answers: 0

Re: ASTER Data Download Cygwin Unable to Retrieve Data

by srleone9 » Sun Jan 02, 2022 12:00 pm America/New_York

Cole, I appreciate your reply and links. I am not a programmer, however, which is why I was looking for guidance with Cygwin in the first place.

Earthdata's very brief instructions - screenshot attached - were to download the .sh file with the 45,824 file names. (It did not specify where to save it, however.)
Then, in the Cygwin command line, enter
chmod 777 download.sh
(From my uneducated research, that allows all users to use the download script.)
Then, to execute the download, enter
./download.sh
(In my further, yet still uneducated, research, there can be "unexpected errors" that result from trying to path to a Windows file. Something about the direction of the slashes?)

I have a feeling that it's a pathing issue, but I don't know how to alter the Cygwin code to make to find the .sh file that I downloaded from Earthdata.
============================

I did try to use CURL (your first link), which is already installed with Windows 10, but I have no idea if it was successful or not. (Remember, I'm not a programmer.) Screenshot attached.

In your reply, you said that I "should be able to feed that text file (using the instructions provided for each of the options) into any of the options listed above", but the template .netrc file is just 3 lines, and there is no guidance on the page you linked to about how to tell CURL which files to download once it logs in. And, again, I don't know enough of the coding syntax to figure out where in those curl commands to "feed that text file" into it. (Although, I finally figured out that I had to type "curl " before all of the lines of code in those instructions.)

I don't know Python, although a beta version is available in the Microsoft Store.
I don't know R, although I have heard of it.

Any more details for us Regular Citizens trying to download topo data?

Thanks,
Su
Attachments
Screenshot of my command prompt
Screenshot of my command prompt
curl-results.png (22.31 KiB) Not viewed yet
This is what Earthdata said to do with Cygwin on a WIndows platform.
This is what Earthdata said to do with Cygwin on a WIndows platform.
Cygwin-Instructions.png (10.47 KiB) Not viewed yet

OB.DAAC - amscott
User Services
User Services
Posts: 249
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 2 times
Been thanked: 1 time

Re: ASTER Data Download Cygwin Unable to Retrieve Data

by OB.DAAC - amscott » Tue Jan 04, 2022 9:29 am America/New_York

Hi Su,

Scientists and programmers take the same approach as the 'regular citizen' should for learning new things. Try typing 'learn wget' or 'learn curl' or 'learn R programming language' into a search engine like Google and begin reading and practicing using those tools. Once you decide which you are more comfortable with, I would suggest revisiting the links Cole shared above to learn how to compose the proper command line statement to retrieve your desired data from the Earthdata Search tool.

Post Reply