Page 1 of 1

Trouble accessing files via S3

Posted: Mon Jan 22, 2024 4:39 am America/New_York
by simonrp1984
I'm trying to download a bunch of MODIS data via a python script using S3 and can't seem to get it working.

This is my script:
import boto3
import s3fs

token = <token from earthdata>
cred_url = ""
header={"Authorization": f"Bearer {token}"}
temp_creds_req = requests.get(cred_url, headers=header).json()

fs_s3 = s3fs.S3FileSystem(anon=False,
client_kwargs={'region_name': 'us-west-2'})

s3_url = 's3://lp-prod-protected/MOD14.061/MOD14.A2024021.1515.061.2024021212648/MOD14.A2024021.1515.061.2024021212648.hdf'
fp =, mode='rb')

Which gives an error:
ClientError Traceback (most recent call last)
File ~\Anaconda3\envs\main\Lib\site-packages\s3fs\, in _error_wrapper(func, args, kwargs, retries)
112 try:
--> 113 return await func(*args, **kwargs)
114 except S3_RETRYABLE_ERRORS as e:

File ~\Anaconda3\envs\main\Lib\site-packages\aiobotocore\, in AioBaseClient._make_api_call(self, operation_name, api_params)
382 error_class = self.exceptions.from_code(error_code)
--> 383 raise error_class(parsed_response, operation_name)
384 else:

ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Any ideas what I'm doing wrong?

Re: Trouble accessing files via S3

Posted: Mon Jan 22, 2024 9:37 am America/New_York
by LP DAAC - lien
We are sending this to our developers to get there take, as soon as I hear anything I will let you know. This product is also in our on-premises Data Pool:
Also we do have a python script that access the data that way:*ep9n50*_ga*MTE0MTk4NjcyOS4xNzA1NDE0NDk3*_ga_0YWDZEJ295*MTcwNTkzMjU2OS4xMy4xLjE3MDU5MzI1NzcuMC4wLjA.

Re: Trouble accessing files via S3

Posted: Mon Jan 22, 2024 10:07 pm America/New_York
by betolink
There is a Python library that tries to simplify how we access data from NASA called earthaccess, it may help your use case, this would be the code you need:

Code: Select all

import earthaccess


max_granules = 30

# C2271754179-LPCLOUD is the identifier for this MODIS dataset, we could also use the DOI if there is one.

granules = earthaccess.search_data(
    cloud_cover=(0,20), # up to 20% of cloud coverage

files =[0:max_granules], "mod14/")
It simplifies how we authenticate with the added value that this code should run even out of region (us-west-2). Hope this helps.