Python 3 Earthdata downloads

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
earthengine_urs
Posts: 72
Joined: Mon Jan 27, 2020 10:36 am America/New_York
Answers: 0
Has thanked: 3 times
Been thanked: 1 time

Python 3 Earthdata downloads

by earthengine_urs » Fri Aug 19, 2022 10:51 pm America/New_York

The first example from viewtopic.php?t=3433#confirm_external_link-modal can be modified for Python 3 like this:

Code: Select all

    import tempfile
    from http import cookiejar
    from urllib import request
    from urllib.parse import urlencode

    # Create a password manager to deal with the 401 response that is returned from
    # Earthdata Login
    password_manager = request.HTTPPasswordMgrWithDefaultRealm()
    password_manager.add_password(None, "https://urs.earthdata.nasa.gov", 'username', 'password')
    # Create a cookie jar for storing cookies. This is used to store and return
    # the session cookie given to use by the data server (otherwise it will just
    # keep sending us back to Earthdata Login to authenticate). Ideally, we
    # should use a file based cookie jar to preserve cookies between runs. This
    # will make it much more efficient.
    cookie_jar = cookiejar.CookieJar()
    # Install all the handlers.
    opener = request.build_opener(
       request.HTTPBasicAuthHandler(password_manager),
       request.HTTPCookieProcessor(cookie_jar))
    request.install_opener(opener)
    # Create and submit the request. There are a wide range of exceptions that
    # can be thrown here, including HTTPError and URLError. These should be
    # caught and handled.
    url = 'https://n5eil01u.ecs.nsidc.org/MOSA/MYD10_L2.061/2002.07.04/MYD10_L2.A2002185.2040.061.2020071133128.hdf'
    myrequest = request.Request(url)
    response = request.urlopen(myrequest)

    # Check if status is OK
    if response.code != 200:
      raise ValueError
    response.begin()
    local_output = tempfile.NamedTemporaryFile(delete=False)
    while True:
      chunk = response.read()
      if chunk: 
        local_output.file.write(chunk)
      else:
        break
    local_output.file.close()
    print(local_output.name)

Tags:

earthengine_urs
Posts: 72
Joined: Mon Jan 27, 2020 10:36 am America/New_York
Answers: 0
Has thanked: 3 times
Been thanked: 1 time

Re: Python 3 Earthdata downloads

by earthengine_urs » Sat Aug 20, 2022 11:56 pm America/New_York

I think I know why some EarthData downloads stopped working. NASA download URLs get redirected to the EarthData login server which requires an authentication header. Setting this header for the original download URL used to be ok, but now it causes a 401 error, and the redirect never happens.

The fix is to only set the authentication header on the second request.

Post Reply