Client Error Unauthorized for GPM_3IMERGHHE

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
jmargetts
Posts: 6
Joined: Tue Feb 07, 2023 5:18 am America/New_York
Answers: 0

Re: Client Error Unauthorized for GPM_3IMERGHHE

by jmargetts » Mon Feb 13, 2023 4:10 am America/New_York

I've double checked my python version and it's python 3.10.4
Attachments
Capture.PNG
Capture.PNG (7.63 KiB) Not viewed yet

Tags:

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Client Error Unauthorized for GPM_3IMERGHHE

by GES DISC - mgreene » Mon Feb 13, 2023 3:36 pm America/New_York

Please see if you can download
https://gpm1.gesdisc.eosdis.nasa.gov/data/GPM_L3/GPM_3IMERGHHL.06/2023/001/3B-HHR-L.MS.MRG.3IMERG.20230101-S000000-E002959.0000.V06C.HDF5 from your Web Browser. Once downloaded please verify that you actually downloaded a file with the data.

If successful, then check your credentials in .netrc file, remove ~/.urs_cookies, create a new empty one and try again. Also verify that wget version is 1.18 or greater.

accuweather_data_partnerships
Posts: 4
Joined: Fri Feb 10, 2023 3:40 pm America/New_York
Answers: 0

Re: Client Error Unauthorized for GPM_3IMERGHHE

by accuweather_data_partnerships » Mon Feb 13, 2023 6:36 pm America/New_York

Hello,

I have done some testing and it appears the redirect from gpm1.gesdisc.eosdis.nasa.gov to urs.earthdata.nasa.gov is no longer occurring before authentication. In my case, and I assume this is the issue with others, the 401 Authentication Error is coming from gpm1.gesdisc.eosdis.nasa.gov because it is no longer redirecting to urs.earthdata.nasa.gov before authenticating. I have verified in my logs that this is what is occurring when attempting to retrieve data via HTTPS using our GoAnywhere MFT Service.

"The request made to '/opendap/hyrax/GPM_L3/GPM_3IMERGHHL.06/2023/032/3B-HHR-L.MS.MRG.3IMERG.20230201-S220000-E222959.1320.V06C.HDF5.nc4' completed with the status 'HTTP/1.1 401 Unauthorized'"

Versus previous behavior (Prior to February 1st) of

The request made to '/opendap/hyrax/GPM_L3/GPM_3IMERGHHL.06/2023/032/3B-HHR-L.MS.MRG.3IMERG.20230201-S210000-E212959.1260.V06C.HDF5.nc4' completed with the status 'HTTP/1.1 302 Found'
Redirecting to 'https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=https%3A%2F%2Fgpm1.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9ncG0xLmdlc2Rpc2MuZW9zZGlzLm5hc2EuZ292L29wZW5kYXAvaHlyYXgvR1BNX0wzL0dQTV8zSU1FUkdISEwuMDYvMjAyMy8wMzIvM0ItSEhSLUwuTVMuTVJHLjNJTUVSRy4yMDIzMDIwMS1TMjEwMDAwLUUyMTI5NTkuMTI2MC5WMDZDLkhERjUubmM0'
Redirecting to 'https://gpm1.gesdisc.eosdis.nasa.gov/data-redirect?code=53813be52cff1803557952669858ba49f3bff1edf7d586cbb17ec0b7f3c12a64&state=aHR0cHM6Ly9ncG0xLmdlc2Rpc2MuZW9zZGlzLm5hc2EuZ292L29wZW5kYXAvaHlyYXgvR1BNX0wzL0dQTV8zSU1FUkdISEwuMDYvMjAyMy8wMzIvM0ItSEhSLUwuTVMuTVJHLjNJTUVSRy4yMDIzMDIwMS1TMjEwMDAwLUUyMTI5NTkuMTI2MC5WMDZDLkhERjUubmM0'
Redirecting to 'https://gpm1.gesdisc.eosdis.nasa.gov/opendap/hyrax/GPM_L3/GPM_3IMERGHHL.06/2023/032/3B-HHR-L.MS.MRG.3IMERG.20230201-S210000-E212959.1260.V06C.HDF5.nc4'

accuweather_data_partnerships
Posts: 4
Joined: Fri Feb 10, 2023 3:40 pm America/New_York
Answers: 0

Re: Client Error Unauthorized for GPM_3IMERGHHE

by accuweather_data_partnerships » Fri Feb 17, 2023 10:39 am America/New_York

Hello,

Has there been any updates in resolving this issue? Our production process has been running for year without issue and we have been without data for 16 days now.

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Client Error Unauthorized for GPM_3IMERGHHE

by GES DISC - mgreene » Fri Feb 17, 2023 1:52 pm America/New_York

I am checking on the status and will be in contact with you soon.

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Client Error Unauthorized for GPM_3IMERGHHE

by GES DISC - mgreene » Fri Feb 17, 2023 2:56 pm America/New_York

To user accuweather_data_partnerships:

Our logs indicate that you had some successful downloads as recently as on
February 13. The logs indicate that these downloads were done using Python
2.27.1, and a browser in Windows. But there are failures, too, using same
Python 2.27.1. It may be true your system does have Python 3, but your download
scripts are Not using your installation of Python 3. Most of the failures
indicate usage of "GoAnywhere/7.1.2", though. Wrt "GoAnywhere", for us it's
impossible to tell if it is using compatible SSL, and we don't recommend using
it in the first place. Please, make sure your system is using your Python 3
installation, and follow carefully the recommendations at:
https://disc.gsfc.nasa.gov/data-access

Logs also indicate failed attempts to use Wget 1.19 in Windows. These are
likely resulting from improper forwarding of your Earthdata credentials in wget
for Windows, which is more peculiar. Please, consult the guidance in the above
data-access web page.

accuweather_data_partnerships
Posts: 4
Joined: Fri Feb 10, 2023 3:40 pm America/New_York
Answers: 0

Re: Client Error Unauthorized for GPM_3IMERGHHE

by accuweather_data_partnerships » Fri Feb 17, 2023 3:36 pm America/New_York

Hello,

Thank you for the quick reply!
The successful downloads via the web browser were to verify that we were using the correct username and password. The downloads using the Python script and wget were a test to diagnose the HTTP 401 errors and see if we could still download the data programmatically.

Our GoAnywhere instance has an option to change the SSL Context Protocol it uses to connect. It defaults to SSL (which is what it has been using for the past year). If the compatible SSL context was updated, we can update our systems to use the correct SSL protocol as the following are compatible (SSL, SSLv2, SSLv3, TLS, TLSv1, TLSv1.1, TLSv1.2)

Can you please clarify if the authentication process or SSL context was updated and what it now is?

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Client Error Unauthorized for GPM_3IMERGHHE

by GES DISC - mgreene » Fri Feb 17, 2023 4:41 pm America/New_York

The data-access page recommends:
gnuTLS 3.3.3 or OpenSSL 1.0.2 or LibreSSL 2.0.2 or later.

Authentication process hasn't been changed.

weather.ingestion_accounts0
Posts: 1
Joined: Mon Jun 22, 2020 12:46 pm America/New_York
Answers: 0

Re: Client Error Unauthorized for GPM_3IMERGHHE

by weather.ingestion_accounts0 » Wed Feb 22, 2023 6:09 pm America/New_York

We also started getting auth failures, I think one of the servers we get redirected to doesn't like being sent auth headers if it didn't request them. We use aiohttp and I've come up with this fix which seems to work:

class ClientRequest(aiohttp.ClientRequest):
def update_auth(self, auth: Optional[aiohttp.BasicAuth]) -> None:
if auth is None:
auth = self.auth

# only send auth if the previous response was a 401
if auth is None or not response_history or response_history[-1].status != 401:
return

self.headers[aiohttp.hdrs.AUTHORIZATION] = auth.encode()

async def send(self, conn: "aiohttp.connector.Connection") -> "ClientResponse":
resp = await super().send(conn)
response_history.append(resp)
return resp

self._client = await self._exit_stack.enter_async_context(aiohttp.ClientSession(
request_class=ClientRequest, auth=self._auth
))

async with self._client.get(url) as response:
if response.status == 401:
await response.read() # so we can re-use connector

# aiohttp won't re-try a 401 with auth by default
if response.status == 401:
async with self._client.get(response.url) as response:
self._logger.info(f"HttpTransfer auth response: {response.status}")

# session is now primed

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Client Error Unauthorized for GPM_3IMERGHHE

by GES DISC - mgreene » Tue Mar 28, 2023 3:49 pm America/New_York

Following up to see if all users on this post were able to download their data of need.

Post Reply