An error occurred (403) when calling the HeadObject operation: Forbidden

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
nerdherdwa
Posts: 3
Joined: Tue Jun 27, 2023 11:33 pm America/New_York
Answers: 0

An error occurred (403) when calling the HeadObject operation: Forbidden

by nerdherdwa » Wed Jun 28, 2023 1:10 am America/New_York

import boto3
import requests

user = 'someusername'
password = 'somepassword
url = 'https://archive.podaac.earthdata.nasa.gov/s3credentials'
url = requests.get(url, allow_redirects=False).headers['Location']
creds = requests.get(url, auth=(user, password)).json()

# creds returned containing sessionToken, secretAcccessKey and accessKeyId

aws_access_key_id = creds['accessKeyId']
aws_secret_access_key = creds['secretAccessKey']
aws_session_token = creds['sessionToken']
region = 'us-west-2'

# pass the retrieve session details to boto3 session
session = boto3.Session(aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
aws_session_token=aws_session_token,
region_name=region
)
client = session.client('s3', verify=False)
bucket = 'podaac-ops-cumulus-protected'
prefix = ''
delimiter = '/'

key = "OSTIA-UKMO-L4-GLOB-v2.0/20230626120000-UKMO-L4_GHRSST-SSTfnd-OSTIA-GLOB-v02.0-fv02.0.nc"
filename = '2023062612-GHRSST-OSTIA.nc'

print('downloading file starting....')
client.download_file(Bucket=bucket, Key=key, Filename=filename)
print('downloading file complete....')

This code executes up until the client download_file line which throws

Traceback (most recent call last):
File "test.py", line 46, in <module>
client.download_file(Bucket=bucket, Key=key, Filename=filename)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

I have checked my aws region that the code is running in and is set to us-west-1.

I have seen other questions around this same issue but with no resolution to how to direct download the data from s3 bucket. All the threads i have seen around the place talk about a misconfigured s3 bucket.

Filters:

PODAAC - jmcnelis
Subject Matter Expert
Subject Matter Expert
Posts: 16
Joined: Tue Mar 14, 2023 1:41 pm America/New_York
Answers: 0

Re: An error occurred (403) when calling the HeadObject operation: Forbidden

by PODAAC - jmcnelis » Wed Jun 28, 2023 1:13 pm America/New_York

Please try again in us-west-2. Thanks!

nerdherdwa
Posts: 3
Joined: Tue Jun 27, 2023 11:33 pm America/New_York
Answers: 0

Re: An error occurred (403) when calling the HeadObject operation: Forbidden

by nerdherdwa » Wed Jun 28, 2023 10:13 pm America/New_York

jmcnelis same response to the previous attempts

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

what versions of python and boto3 are expected, currently running version 3.6.9 of python and boto3==1.16.56
botocore==1.19.63

PODAAC - jmcnelis
Subject Matter Expert
Subject Matter Expert
Posts: 16
Joined: Tue Mar 14, 2023 1:41 pm America/New_York
Answers: 0

Re: An error occurred (403) when calling the HeadObject operation: Forbidden

by PODAAC - jmcnelis » Thu Jun 29, 2023 2:28 pm America/New_York

Try s3fs as described here and let me know if that doesn't work for you: https://podaac.github.io/tutorials/external/Direct_S3_Access_NetCDF.html

Thanks,
Jack

nerdherdwa
Posts: 3
Joined: Tue Jun 27, 2023 11:33 pm America/New_York
Answers: 0

Re: An error occurred (403) when calling the HeadObject operation: Forbidden

by nerdherdwa » Thu Jun 29, 2023 11:20 pm America/New_York

Thanks Jack have tried that with the same resultant error though the stack trace was further enhanced with the following information

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

PermissionError: Forbidden

PODAAC - jmcnelis
Subject Matter Expert
Subject Matter Expert
Posts: 16
Joined: Tue Mar 14, 2023 1:41 pm America/New_York
Answers: 0

Re: An error occurred (403) when calling the HeadObject operation: Forbidden

by PODAAC - jmcnelis » Thu Aug 17, 2023 5:04 pm America/New_York

Hi nerdherdwa,

Please issue the following command in your shell and report back with the output:

aws configure get region

Thanks,
Jack

fmis
Posts: 1
Joined: Thu Aug 17, 2023 10:11 pm America/New_York
Answers: 0

Re: An error occurred (403) when calling the HeadObject operation: Forbidden

by fmis » Thu Aug 17, 2023 10:32 pm America/New_York

us-west-2 is the output from the console. which is being set by both the code and is configured using the aws cli.

session = boto3.Session(
aws_access_key_id=creds["accessKeyId"],
aws_secret_access_key=creds["secretAccessKey"],
aws_session_token=creds["sessionToken"],
region_name='us-west-2'
)

boto should only be using the aws cli configured region when one isnt provided as part of the session as per the code above. At this point the S3 boto doesnt look viable and have reverted to a https request for the GHRSST-OSTIA files passing headers in with the authentication session. Preference would be for a S3 download but i dont think the configuration/permissions of the S3 buckets have been setup correctly to allow this type of download or there is an issue with boto when it passes in the session information. My code works with S3 buckets that i have in my own AWS environment

PODAAC - jmcnelis
Subject Matter Expert
Subject Matter Expert
Posts: 16
Joined: Tue Mar 14, 2023 1:41 pm America/New_York
Answers: 0

Re: An error occurred (403) when calling the HeadObject operation: Forbidden

by PODAAC - jmcnelis » Tue Sep 05, 2023 1:59 pm America/New_York

Thanks nerdherdwa. I can't offer any explanation for why access through boto3 is not working from your EC2 instance running in us-west-2.

The s3fs approach works for me against this same collection. We are looking into it; can you use s3fs instead in the meantime? Here's my working code:
import os
import s3fs
import requests
import xarray as xr

def begin_s3_direct_access(url: str="https://archive.podaac.earthdata.nasa.gov/s3credentials"):
response = requests.get(url).json()
return s3fs.S3FileSystem(key=response['accessKeyId'],
secret=response['secretAccessKey'],
token=response['sessionToken'],
client_kwargs={'region_name':'us-west-2'})

fs = begin_s3_direct_access()

short_name = "OSTIA-UKMO-L4-GLOB-v2.0"

files = pd.Series(sorted(fs.glob(os.path.join("podaac-ops-cumulus-protected/", short_name, "*.nc"))))
This code assumes you have Earthdata Login credentials set up properly inside your .netrc file.

Thanks for bringing the issue to our attention.

Jack

Post Reply