Request's response 502 Gateway error with limit set to 250

[mshahbaz_000]

Through a python script, I am sending a POST request to retrieve items from LPCLOUD STAC Catalog. My request parameters are as follows:

params = {'limit': 250, 'bbox': '-112.63573388436109,56.74522108547581,-110.83605841365043,57.74219208744419', 'datetime': '2023-01-02T00:00:00Z/2023-11-09T23:59:59Z', 'collections': ['HLSS30.v2.0']}

The POST command is like this:
items = requests.post(lp_search, json=params)

with lp_search being the search endpoint for the LPCLOUD STAC Catalog:
https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search

This results in response status code of 502 (bad Gateway server error). The way I can fix it is by reducing the 'limit' to 100.

I am wondering why the error happens with larger limits? In some requests (e.g. with a much shorter datetime range and a much smaller bbox), the limit of 250 matched items works fine. But, in some requests like the one above, a smaller limit should be used. Is there a safe way to determine this limit so that we would not get an unexpected response error?

[LP DAAC - dgolon]

Hi @mshahbaz_000 Let me check with our developers and we'll get back to you. Thanks -- Danielle

[LP DAAC - kyrad]

@mshahbaz_000,
There are no limitations for the number of granules to request through the STAC catalog however response times can differ between datasets. The 502 error is due to a timing out issue because the requests are taking a long time to download and the network on the backend is closing. We recommend breaking down the requests into smaller segments that way the downloading is quicker. 

If interested in, we have a tutorial on utilizing the CMR-STAC API https://git.earthdata.nasa.gov/projects/LPDUR/repos/data-discovery---cmr-stac-api-in-python/browse/notebooks/Data%20Discovery%20-%20CMR-STAC%20API.ipynb 

If you continue to have issues, please send us your entire script and we can try to replicate the problem.
Thanks!

[mshahbaz_000]

Thank you very much for your response. I will then make sure the requests are broken to small segments.