Unable to acces GES DISC DATA ARCHIVE from AWS S#

[gabriel.ferrari.bt]

Hello everyone,

I'm facing difficulties accessing data from the S3 bucket in the us-west-2 zone. I've diligently followed the steps from the AWS official documentation and double-checked multiple times to ensure I'm in the correct zone. Furthermore, I attempted to access the data through an EC2 instance, but I'm still encountering issues.

Here's a snippet of my code:

from netrc import netrc

import os
import requests
import s3fs


urs = 'urs.earthdata.nasa.gov'
netrc_name = ".netrc"
netrcDir = os.path.expanduser(f"~/{netrc_name}")
netrc(netrcDir).authenticators(urs)[0]


gesdisc_s3 = "https://data.gesdisc.earthdata.nasa.gov/s3credentials"


def begin_s3_direct_access(url: str=gesdisc_s3):
response = requests.get(url).json()
return s3fs.S3FileSystem(key=response['accessKeyId'],
secret=response['secretAccessKey'],
token=response['sessionToken'],
client_kwargs={'region_name':'us-west-2'})

fs = begin_s3_direct_access()


fn = 's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGHHE.06/2023/091/3B-HHR-E.MS.MRG.3IMERG.20230401-S000000-E002959.0000.V06C.HDF5'

fs.ls(fn)


However, upon running this code, I receive the following error message:

---------------------------------------------------------------------------
ClientError Traceback (most recent call last)
File ~/notebooks/dsfc_nasa/.venv/lib/python3.10/site-packages/s3fs/core.py:113, in _error_wrapper(func, args, kwargs, retries)
112 try:
--> 113 return await func(*args, **kwargs)
114 except S3_RETRYABLE_ERRORS as e:

File ~/notebooks/dsfc_nasa/.venv/lib/python3.10/site-packages/aiobotocore/client.py:383, in AioBaseClient._make_api_call(self, operation_name, api_params)
382 error_class = self.exceptions.from_code(error_code)
--> 383 raise error_class(parsed_response, operation_name)
384 else:

ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

PermissionError Traceback (most recent call last)
/home/gabriel/notebooks/dsfc_nasa/open.py in line 38
35 # Files are organized by s3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/year/mo/*.nc4
37 fn = 's3://gesdisc-cumulus-prod-protected/GPM_L3/GPM_3IMERGHHE.06/2023/091/3B-HHR-E.MS.MRG.3IMERG.20230401-S000000-E002959.0000.V06C.HDF5'
---> 38 fs.info(fn)

File ~/notebooks/dsfc_nasa/.venv/lib/python3.10/site-packages/fsspec/asyn.py:118, in sync_wrapper.<locals>.wrapper(*args, **kwargs)
115 @functools.wraps(func)
116 def wrapper(*args, **kwargs):
...
138 err = e
139 err = translate_boto_error(err)
--> 140 raise err

PermissionError: Forbidden

I'd greatly appreciate any guidance or suggestions to resolve this issue. I've reviewed my permissions in the site, followed the steps outlined in the documentation, and am confident I'm in the correct zone.

Thank you.

[cbattisto]

Hello,

My name is Chris Battisto, a User Needs team member at the GES DISC. I'm sorry to hear you're experiencing this error, and I can help you troubleshoot.

Yes, this error is usually experienced when the code is being run out-of-region. To further verify that you are in the correct region, could you please run this Python code inside of the EC2 instance, and verify that the output is "us-west-2"?

import boto3
boto3.client('s3').meta.region_name

Thank you!
Chris

[cbattisto]

Hello,

Just checking in about this issue. Have you been able to resolve the error?

Chris

[bharatsharma]

To practically address the ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden error in AWS, you can follow a step-by-step approach to troubleshoot and resolve the issue. Here's a practical guide:
1. Verify IAM Permissions
Action: Check the IAM (Identity and Access Management) policies attached to the user or role you're using.
How:
Go to the AWS IAM console.
Find the user or role used for accessing the S3 bucket.
Ensure it has s3:GetObject permission for the specific bucket (gesdisc-cumulus-prod-protected).
2. Check Bucket Policy and Access Control List (ACL)
Action: Verify the S3 bucket's policy and ACL to ensure they allow access to your IAM user or role.
How:
Go to the S3 console.
Select the bucket and check the "Permissions" tab.
Look for any statements that might be denying access or not including your user/role.
3. Confirm Region and Endpoint
Action: Ensure that the bucket is indeed in us-west-2 and that you're connecting to the correct endpoint.
How:
In the S3 console, check the bucket's region.
In your code, ensure client_kwargs={'region_name':'us-west-2'} matches the bucket's region.
4. Validate Temporary Credentials
Action: Make sure the temporary credentials obtained from GES DISC are valid and not expired.
How:
Examine the response from https://data.gesdisc.earthdata.nasa.gov/s3credentials.
Check the expiry time of the sessionToken.
5. Object Path Verification
Action: Double-check the S3 object path for typos or errors.
How:
Verify the object's key (fn) matches exactly with what's stored in S3, including case sensitivity.
6. Use AWS CLI for Debugging
Action: Use the AWS Command Line Interface to attempt accessing the file.
How:
Run aws s3 ls s3://gesdisc-cumulus-prod-protected/... using the same credentials.
Look for any additional error messages or details.
7. Network and Firewall Configuration
Action: Check if there are network restrictions like VPC endpoints or firewalls blocking access.
How:
Review your network ACLs, security groups, and any VPC endpoint configurations.
8. Seek Support
Action: If the issue persists, contact AWS Support or GES DISC for assistance.
How:
Open a support ticket through the AWS Management Console or contact GES DISC support.
Additional Tips
Logging: Enable logging on the S3 bucket to track access requests and failures.
Documentation: Refer to AWS documentation for specific guidelines on S3 access and troubleshooting.
Community Forums: Sometimes AWS forums or communities like Stack Overflow can provide insights based on similar experiences.
Remember, the key is to systematically check each aspect, starting from IAM permissions to network configurations. Often, such access issues are related to permissions or misconfigurations.

[gabriel.ferrari.bt]

Hi Chris,

Thank you very much and sorry for the delay in responding. I just checked the aws_region on the EC2 machine located in the us-west-2 region, but it was set to the default region for my account, which was us-east-1. The problem was solved.