Earthdata Forum

I've double checked my python version and it's python 3.10.4

Please see if you can download
https://gpm1.gesdisc.eosdis.nasa.gov/data/GPM_L3/GPM_3IMERGHHL.06/2023/001/3B-HHR-L.MS.MRG.3IMERG.20230101-S000000-E002959.0000.V06C.HDF5 from your Web Browser. Once downloaded please verify that you actually downloaded a file with the data.

If successful, then check your credentials in .netrc file, remove ~/.urs_cookies, create a new empty one and try again. Also verify that wget version is 1.18 or greater.

Hello,

I have done some testing and it appears the redirect from gpm1.gesdisc.eosdis.nasa.gov to urs.earthdata.nasa.gov is no longer occurring before authentication. In my case, and I assume this is the issue with others, the 401 Authentication Error is coming from gpm1.gesdisc.eosdis.nasa.gov because it is no longer redirecting to urs.earthdata.nasa.gov before authenticating. I have verified in my logs that this is what is occurring when attempting to retrieve data via HTTPS using our GoAnywhere MFT Service.

"The request made to '/opendap/hyrax/GPM_L3/GPM_3IMERGHHL.06/2023/032/3B-HHR-L.MS.MRG.3IMERG.20230201-S220000-E222959.1320.V06C.HDF5.nc4' completed with the status 'HTTP/1.1 401 Unauthorized'"

Versus previous behavior (Prior to February 1st) of

The request made to '/opendap/hyrax/GPM_L3/GPM_3IMERGHHL.06/2023/032/3B-HHR-L.MS.MRG.3IMERG.20230201-S210000-E212959.1260.V06C.HDF5.nc4' completed with the status 'HTTP/1.1 302 Found'
Redirecting to 'https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=https%3A%2F%2Fgpm1.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9ncG0xLmdlc2Rpc2MuZW9zZGlzLm5hc2EuZ292L29wZW5kYXAvaHlyYXgvR1BNX0wzL0dQTV8zSU1FUkdISEwuMDYvMjAyMy8wMzIvM0ItSEhSLUwuTVMuTVJHLjNJTUVSRy4yMDIzMDIwMS1TMjEwMDAwLUUyMTI5NTkuMTI2MC5WMDZDLkhERjUubmM0'
Redirecting to 'https://gpm1.gesdisc.eosdis.nasa.gov/data-redirect?code=53813be52cff1803557952669858ba49f3bff1edf7d586cbb17ec0b7f3c12a64&state=aHR0cHM6Ly9ncG0xLmdlc2Rpc2MuZW9zZGlzLm5hc2EuZ292L29wZW5kYXAvaHlyYXgvR1BNX0wzL0dQTV8zSU1FUkdISEwuMDYvMjAyMy8wMzIvM0ItSEhSLUwuTVMuTVJHLjNJTUVSRy4yMDIzMDIwMS1TMjEwMDAwLUUyMTI5NTkuMTI2MC5WMDZDLkhERjUubmM0'
Redirecting to 'https://gpm1.gesdisc.eosdis.nasa.gov/opendap/hyrax/GPM_L3/GPM_3IMERGHHL.06/2023/032/3B-HHR-L.MS.MRG.3IMERG.20230201-S210000-E212959.1260.V06C.HDF5.nc4'

Hello,

Has there been any updates in resolving this issue? Our production process has been running for year without issue and we have been without data for 16 days now.

I am checking on the status and will be in contact with you soon.

To user accuweather_data_partnerships:

Our logs indicate that you had some successful downloads as recently as on
February 13. The logs indicate that these downloads were done using Python
2.27.1, and a browser in Windows. But there are failures, too, using same
Python 2.27.1. It may be true your system does have Python 3, but your download
scripts are Not using your installation of Python 3. Most of the failures
indicate usage of "GoAnywhere/7.1.2", though. Wrt "GoAnywhere", for us it's
impossible to tell if it is using compatible SSL, and we don't recommend using
it in the first place. Please, make sure your system is using your Python 3
installation, and follow carefully the recommendations at:
https://disc.gsfc.nasa.gov/data-access

Logs also indicate failed attempts to use Wget 1.19 in Windows. These are
likely resulting from improper forwarding of your Earthdata credentials in wget
for Windows, which is more peculiar. Please, consult the guidance in the above
data-access web page.

Hello,

Thank you for the quick reply!
The successful downloads via the web browser were to verify that we were using the correct username and password. The downloads using the Python script and wget were a test to diagnose the HTTP 401 errors and see if we could still download the data programmatically.

Our GoAnywhere instance has an option to change the SSL Context Protocol it uses to connect. It defaults to SSL (which is what it has been using for the past year). If the compatible SSL context was updated, we can update our systems to use the correct SSL protocol as the following are compatible (SSL, SSLv2, SSLv3, TLS, TLSv1, TLSv1.1, TLSv1.2)

Can you please clarify if the authentication process or SSL context was updated and what it now is?

The data-access page recommends:
gnuTLS 3.3.3 or OpenSSL 1.0.2 or LibreSSL 2.0.2 or later.

Authentication process hasn't been changed.

We also started getting auth failures, I think one of the servers we get redirected to doesn't like being sent auth headers if it didn't request them. We use aiohttp and I've come up with this fix which seems to work:

class ClientRequest(aiohttp.ClientRequest):
def update_auth(self, auth: Optional[aiohttp.BasicAuth]) -> None:
if auth is None:
auth = self.auth

# only send auth if the previous response was a 401
if auth is None or not response_history or response_history[-1].status != 401:
return

self.headers[aiohttp.hdrs.AUTHORIZATION] = auth.encode()

async def send(self, conn: "aiohttp.connector.Connection") -> "ClientResponse":
resp = await super().send(conn)
response_history.append(resp)
return resp

self._client = await self._exit_stack.enter_async_context(aiohttp.ClientSession(
request_class=ClientRequest, auth=self._auth
))

async with self._client.get(url) as response:
if response.status == 401:
await response.read() # so we can re-use connector

# aiohttp won't re-try a 401 with auth by default
if response.status == 401:
async with self._client.get(response.url) as response:
self._logger.info(f"HttpTransfer auth response: {response.status}")

# session is now primed

Following up to see if all users on this post were able to download their data of need.