The problem: the workers don't share a credential cache and don't go through a broker, so on scale-out (or after credentials expire) each new execution environment independently hits the DAAC's s3credentials endpoint to mint its own temporary credentials. Since the credentials are bucket-scoped and effectively identical, this is redundant and it produces a burst of credential requests against the STS endpoint within a short window. We've been calling this a "credential stampede" or "thundering herd on cold start."
I know this is a commonly encountered pattern for applications reading EO data at scale, so before we design around it I wanted to ask:
- Is there existing guidance, a recommended pattern, or a reference implementation for acquiring DAAC S3 temporary credentials in a highly concurrent, brokerless serverless environment?
- Is there a preferred approach on the Earthdata side, such as a shared credential endpoint, a token/credential caching service, or guidance on expected request rates to STS so we can size our refresh strategy?
Any pointers would be appreciated.