Trouble accessing files via S3

[simonrp1984]

Hello,
I'm trying to download a bunch of MODIS data via a python script using S3 and can't seem to get it working.

This is my script:
```
import boto3
import s3fs

token = <token from earthdata>
cred_url = "https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials"
header={"Authorization": f"Bearer {token}"}
temp_creds_req = requests.get(cred_url, headers=header).json()

fs_s3 = s3fs.S3FileSystem(anon=False,
key=temp_creds_req['accessKeyId'],
secret=temp_creds_req['secretAccessKey'],
token=temp_creds_req['sessionToken'],
client_kwargs={'region_name': 'us-west-2'})

s3_url = 's3://lp-prod-protected/MOD14.061/MOD14.A2024021.1515.061.2024021212648/MOD14.A2024021.1515.061.2024021212648.hdf'
fp = fs_s3.open(s3_url, mode='rb')
```

Which gives an error:
```
---------------------------------------------------------------------------
ClientError Traceback (most recent call last)
File ~\Anaconda3\envs\main\Lib\site-packages\s3fs\core.py:113, in _error_wrapper(func, args, kwargs, retries)
112 try:
--> 113 return await func(*args, **kwargs)
114 except S3_RETRYABLE_ERRORS as e:

File ~\Anaconda3\envs\main\Lib\site-packages\aiobotocore\client.py:383, in AioBaseClient._make_api_call(self, operation_name, api_params)
382 error_class = self.exceptions.from_code(error_code)
--> 383 raise error_class(parsed_response, operation_name)
384 else:

ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
```
Any ideas what I'm doing wrong?

[LP DAACx - lien]

Hello,
We are sending this to our developers to get there take, as soon as I hear anything I will let you know. This product is also in our on-premises Data Pool: https://e4ftl01.cr.usgs.gov/MOLT/MOD14.061/
Also we do have a python script that access the data that way: https://git.earthdata.nasa.gov/projects/LPDUR/repos/daac_data_download_python/browse?_gl=1*ep9n50*_ga*MTE0MTk4NjcyOS4xNzA1NDE0NDk3*_ga_0YWDZEJ295*MTcwNTkzMjU2OS4xMy4xLjE3MDU5MzI1NzcuMC4wLjA.
Thanks,
Brett

[betolink]

There is a Python library that tries to simplify how we access data from NASA called earthaccess https://github.com/nsidc/earthaccess, it may help your use case, this would be the code you need:

Code: Select all

import earthaccess

earthaccess.login()

max_granules = 30

# C2271754179-LPCLOUD is the identifier for this MODIS dataset, we could also use the DOI if there is one.

granules = earthaccess.search_data(
    concept_id="C2271754179-LPCLOUD",
    temporal=("2023-01","2023-12"),
    bounding_box=(-107.40,1.23,-73.30,27.52),
    cloud_cover=(0,20), # up to 20% of cloud coverage
    count=max_granules
)

files = earthaccess.download(granules[0:max_granules], "mod14/")

It simplifies how we authenticate with the added value that this code should run even out of region (us-west-2). Hope this helps.