Access GRACE data from Jupyter Notebook

Please enter here to ask a question about any NASA Science related topics!
Post Reply
aaschwanden
Posts: 1
Joined: Wed Apr 07, 2021 11:36 am America/New_York

Access GRACE data from Jupyter Notebook

by aaschwanden » Wed Apr 07, 2021 11:42 am America/New_York

I’ve been trying for a while now to access the following data set to be used inside a Jupyter notebook

Short Name : GREENLAND_MASS_TELLUS_MASCON_CRI_TIME_SERIES_RL06_V2
URL : https://podaac-tools.jpl.nasa.gov/drive ... 202101.txt

So far, I tried the PO.DAAC python module, as well as different authentication methods. I’m attaching a script that test two methods. My credentials are saved in a .netrc file, and the web-based access works just fine.

The first method is based on https://github.com/podaac/AGU-2020 cloud-access-analysis.ipynb. Interestingly, I get a response of 200, which should indicated success, but the data it downloads is still HTML saying I don’t have access.

The second method creates and installs an opener for the “requests”

This is puzzling me. Ideally, I’d like to stream the data directly into a pandas data frame, without creating a physical file on disk.

Any hints are greatly appreciated.

[code]
import itertools
import netrc
import getpass
import os
import requests
from requests.auth import HTTPDigestAuth
from urllib.parse import urlparse
from urllib.request import HTTPError
from urllib import request
from netrc import netrc
from platform import system
from http.cookiejar import CookieJar
import netrc
from getpass import getpass

URS_URL = "https://urs.earthdata.nasa.gov"


def get_username():
username = ""

# For Python 2/3 compatibility:
try:
do_input = raw_input # noqa
except NameError:
do_input = input

while not username:
try:
username = do_input("Earthdata username: ")
except KeyboardInterrupt:
quit()
return username


def get_password():
password = ""
while not password:
try:
password = getpass.getpass("password: ")
except KeyboardInterrupt:
quit()
return password


def get_credentials():
"""Get user credentials from .netrc or prompt for input."""
credentials = None
errprefix = ""
try:
info = netrc.netrc()
username, account, password = info.authenticators(urlparse(URS_URL).hostname)
errprefix = "netrc error: "
except Exception as e:
if not ("No such file" in str(e)):
print("netrc error: {0}".format(str(e)))
username = None
password = None

if not username:
username = get_username()
password = get_password()

return (username, password)


def download(url, outpath=""):
"""Download files from url."""
if not url:
return

print(f"Downloading {url} files to {outpath}...")
username = None

if not username and urlparse(url).scheme == "https":
username, password = get_credentials()

filename = os.path.join(outpath, url.split("/")[-1])

try:
response = requests.get(url, auth=HTTPDigestAuth(username, password))
response.raise_for_status()
except HTTPError as http_err:
print(f"HTTP Error: {http_error}")
except Exception as err:
print(f"Error: {err}")
except KeyboardInterrupt:
quit()
else:
open(filename, "wb").write(response.content)

print("\n\n")
print(response)
print("\n\n")

return response.content


def setup_earthdata_login_auth(endpoint: str = "urs.earthdata.nasa.gov"):

netrc_name = "_netrc" if system() == "Windows" else ".netrc"
try:
username, password = get_credentials()
except (FileNotFoundError, TypeError):
print("Please provide your Earthdata Login credentials for access.")
print("Your info will only be passed to %s and will not be exposed in Jupyter." % (endpoint))
username = input("Username: ")
password = getpass("Password: ")
manager = request.HTTPPasswordMgrWithDefaultRealm()
manager.add_password(None, endpoint, username, password)
auth = request.HTTPBasicAuthHandler(manager)
jar = CookieJar()
processor = request.HTTPCookieProcessor(jar)
opener = request.build_opener(auth, processor)
request.install_opener(opener)
return request


def test_method_1(url):
"""
Trying with HTTPDigestAuth authentification
"""
r = download(url)
print("\n\n")
print(r)
print("\n\n")


def test_method_2(url):
"""
Trying with earthdata login
"""
request = setup_earthdata_login_auth()
r = request.urlopen(url).read()
print("\n\n")
print(r)
print("\n\n")


url = "https://podaac-tools.jpl.nasa.gov/drive ... 202101.txt"

test_method_1(url)
test_method_2(url)

[/code]

Tags:

CDDIS_support_Lori
User Services
User Services
Posts: 51
Joined: Mon Sep 30, 2019 11:17 am America/New_York

Re: Access GRACE data from Jupyter Notebook

by CDDIS_support_Lori » Tue Jun 22, 2021 12:14 pm America/New_York

Hello,

It seems your question involves data archived at the Physical Oceanography DAAC (podaac). Your question can be asked in the PODAAC forum located at: https://podaac.jpl.nasa.gov/forum/ or by emailing their support staff at podaac@podaac.jpl.nasa.gov.

Post Reply