Earthdata Forum

I'm working on a service that reads Earth observation data from a DAAC S3 bucket. It runs as a horizontally scaled set of AWS Lambda functions with many concurrent execution environments, each handling different data, but all reading from the same bucket and therefore all needing the same bucket-scoped temporary S3 credentials.

The problem: the workers don't share a credential cache and don't go through a broker, so on scale-out (or after credentials expire) each new execution environment independently hits the DAAC's s3credentials endpoint to mint its own temporary credentials. Since the credentials are bucket-scoped and effectively identical, this is redundant and it produces a burst of credential requests against the STS endpoint within a short window. We've been calling this a "credential stampede" or "thundering herd on cold start."

I know this is a commonly encountered pattern for applications reading EO data at scale, so before we design around it I wanted to ask:

1. Is there existing guidance, a recommended pattern, or a reference implementation for acquiring DAAC S3 temporary credentials in a highly concurrent, brokerless serverless environment?
2. Is there a preferred approach on the Earthdata side, such as a shared credential endpoint, a token/credential caching service, or guidance on expected request rates to STS so we can size our refresh strategy?

We are considering various designs, but we'd rather not reinvent something if there's an established solution, and we'd like to stay within whatever rate expectations Earthdata has for STS.

Any pointers would be appreciated.