Unable to access the content of s3://gesdisc-cumulus-prod-protected/ by using s3cmd command

[datam]

Dear support,

Can you help me to access the content?

id@hostname: ~> s3cmd -c ~/.s3cfg_nasa_gesdisc --no-check-certificate ls s3://gesdisc-cumulus-prod-protected/
WARNING: Could not refresh role
ERROR: Access to bucket 'gesdisc-cumulus-prod-protected' was denied
ERROR: S3 error: 403 (AccessDenied): User: arn:aws:sts::383133334080:assumed-role/s3-same-region-access-role/datam is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::gesdisc-cumulus-prod-protected" with an explicit deny in an identity-based policy


The credentials came from https://data.gesdisc.earthdata.nasa.gov/s3credentials

id@host: ~> jq < ~/.s3cfg_nasa_asdc.json
{
"accessKeyId": "ASIAXZ5UE54DOKMRSEOC",
"secretAccessKey": "+vLqD4xf48YCon0V/yIzQF46PMVVH1nxq6dlJVdd",
"sessionToken": "LONG_CHARACTERS_STRING...0oo49AJP3xFby70l3fm0RbZsY0W26266R2ItDZfWWZGqv0wD7KL4g0qWO27vArG",
"expiration": "2026-03-03 09:10:02+00:00"
}


Herebelow is my updated config. file:

id@host: ~> cat ~/.s3cfg_nasa_gesdisc
[default]
access_key = ASIAVSNEGXJAE2DCGIYN
access_token = LONG_CHARACTERS_STRING...0oo49AJP3xFby70l3fm0RbZsY0W26266R2ItDZfWWZGqv0wD7KL4g0qWO27vArG
add_encoding_exts =
add_headers =
bucket_location = us-west-2
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
connection_max_age = 5
connection_pooling = True
content_disposition =
content_type =
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = s3://gesdisc-cumulus-prod-protected
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
keep_dirs = False
kms_key =
limit = -1
limitrate = 0
list_allow_unordered = False
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
max_retries = 5
mime_type =
multipart_chunk_size_mb = 15
multipart_copy_chunk_size_mb = 1024
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
public_url_use_https = False
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
secret_key = Tn5cuS/VBbr22RCy5OQ0JIPaPQiadG7DtoMFjATc
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
signurl_use_https = False
simpledb_host = sdb.amazonaws.com
skip_destination_validation = False
skip_existing = False
socket_timeout = 300
ssl_client_cert_file =
ssl_client_key_file =
stats = False
stop_on_error = False
storage_class =
throttle_max = 100
upload_id =
urlencoding_mode = normal
use_http_expect = False
use_https = True
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html

==========================================================================

For info, note that i can access successfully the content of s3://asdc-prod-protected/:

id@hostname: ~> s3cmd -c ~/.s3cfg_nasa_asdc --no-check-certificate ls s3://asdc-prod-protected/
WARNING: Could not refresh role
DIR s3://asdc-prod-protected/ACRIM_III/
DIR s3://asdc-prod-protected/CALIOP/
DIR s3://asdc-prod-protected/CALIPSO/
DIR s3://asdc-prod-protected/CERES/
DIR s3://asdc-prod-protected/DSCOVR/
DIR s3://asdc-prod-protected/EDOS/
DIR s3://asdc-prod-protected/FIELDCAMPAIGN/
DIR s3://asdc-prod-protected/GSESA/
DIR s3://asdc-prod-protected/LeapSecT.001/
DIR s3://asdc-prod-protected/MISR/
DIR s3://asdc-prod-protected/MOPITT/
DIR s3://asdc-prod-protected/PREFIRE/
DIR s3://asdc-prod-protected/SAGE_III/
DIR s3://asdc-prod-protected/SAGE_III_ISS/
DIR s3://asdc-prod-protected/TEMPO/
DIR s3://asdc-prod-protected/TES/
DIR s3://asdc-prod-protected/browse/

Can you help me please?

[datam]

What do I need to do to access s3://gesdisc-cumulus-prod-protected/, please?

[GES DISC - alouise517]

Hello,

The S3 credentials endpoint is not intended to be used to list all the objects in NASA S3 buckets, and the AWS role specified will not have permission to use ListBucket. Instead, the Common Metadata Repository will need to be queried to retrieve all the S3 URLs for that particular bucket, and the S3 credentials endpoint is used to individually access each of those objects.

Here is a Jupyter Notebook which can retrieve and list S3 URLs with the earthaccess library: https://github.com/nasa/gesdisc-tutorials/blob/main/cloud-tutorials/notebooks/How_to_Obtain_a_List_of_S3_URLs_for_GES_DISC_Collection_Using_Python.ipynb. More information about the CMR API can be found here: https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#c-cloud-hosted.

Thank you for your patience, and please let us know if you need further assistance or encounter additional issues.

Regards,
GES DISC User Services