If I’m working in a different AWS region from where NASA Earthdata is located, what is the best way to freely and ......

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
PODAAC - colleen
User Services
User Services
Posts: 71
Joined: Thu May 27, 2021 2:52 pm America/New_York
Answers: 0

If I’m working in a different AWS region from where NASA Earthdata is located, what is the best way to freely and ......

by PODAAC - colleen » Mon Feb 13, 2023 5:43 pm America/New_York

If I’m working in a different AWS region from where NASA Earthdata is located, what is the best way to freely and easily connect to that data, and also helps with authenticating

User Inquiry:
I'm in the process of moving my scripts from PO.DAAC Drive to Earthdata Cloud, and I'm having a difficult time with data downloads.

From what I can tell, there are two ways to access cloud data: Direct S3 Access and Direct Access through https://archive.podaac.earthdata.nasa.gov.

Direct S3 access is not possible for me because the temporary credentials can only be used in-region, and my existing backend is not set up to run in us-west-2. (That would be a big undertaking.)

This leaves me with direct access via HTTPS. However, I am struggling to find an easy way to download from the URLs with my scripts. It seems oauth is required, using my earthdata username and password, and there are multiple redirects and cookies to track. I thought generating a user token (https://urs.earthdata.nasa.gov/users/cambecc/user_tokens) and specifying that on the requests could allow me to skip oauth, but it seems no. Is there no easier way, or will I just need to use oauth to download the data? Or is there something else I'm missing?

Response to User:
Thanks for your email, we’ve really tried to make the transition to the cloud endpoints easier, but it’s not as simple at all times. Without know much about your scripts, I can offer a few pointers.

1. We have a tool used to download data from PO.DAAC in the cloud that might solve many of your issues, it’s called the data subscriber and can be found here: https://github.com/podaac/data-subscriber it allows you to download data either in bulk or continuously over time and place it on a local file system- very similar to normal download operations.

2. You can use wget and set up your credentials to download data, then it’s as simple as running (note: you’ll need a netrc file for this to work out of the box, using your earthdata username and login):
a. wget https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR25-JPL-L4-GLOB-v04.2/20230207090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2.nc

3. if you want to run programmatically, with something like python, you can set up a few different options to automatically do the cookie saving work for you- we can point you at some code examples if you’d like further info on doing this.

Again, I appreciate your email and hope to resolve your issues as soon as possible! Please follow up with any questions on the above options or if one of them works for your use case!

User Update:
Thanks for the detailed response. I’m not using wget but I am using curl. And I was able to successfully configure both .netrc and the curl command line options to quite easily download the files. curl thankfully takes care of all the redirects and cookie tracking. And the data-subscriber tool was quite useful for helping me understand how to make my own calls to the cmr granule search api with JavaScript. So all is now good.

Thanks again for the help and for all the hard work keeping PO.DAAC up and running!

Tags:

LP - brilind
Subject Matter Expert
Subject Matter Expert
Posts: 1
Joined: Fri Jun 10, 2022 12:24 pm America/New_York
Answers: 0

Re: If I’m working in a different AWS region from where NASA Earthdata is located, what is the best way to freely and ..

by LP - brilind » Mon Apr 03, 2023 9:49 pm America/New_York

Check out this cloud cookbook for more info!
https://nasa-openscapes.github.io/earthdata-cloud-cookbook/

Post Reply