Through a python script, I am sending a POST request to retrieve items from LPCLOUD STAC Catalog. My request parameters are as follows:
params = {'limit': 250, 'bbox': '-112.63573388436109,56.74522108547581,-110.83605841365043,57.74219208744419', 'datetime': '2023-01-02T00:00:00Z/2023-11-09T23:59:59Z', 'collections': ['HLSS30.v2.0']}
The POST command is like this:
items = requests.post(lp_search, json=params)
with lp_search being the search endpoint for the LPCLOUD STAC Catalog:
https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search
This results in response status code of 502 (bad Gateway server error). The way I can fix it is by reducing the 'limit' to 100.
I am wondering why the error happens with larger limits? In some requests (e.g. with a much shorter datetime range and a much smaller bbox), the limit of 250 matched items works fine. But, in some requests like the one above, a smaller limit should be used. Is there a safe way to determine this limit so that we would not get an unexpected response error?
Request's response 502 Gateway error with limit set to 250
-
- Posts: 3
- Joined: Thu Nov 09, 2023 7:11 pm America/New_York
-
- User Services
- Posts: 296
- Joined: Mon Sep 30, 2019 10:00 am America/New_York
- Has thanked: 18 times
- Been thanked: 2 times
- Contact:
Re: Request's response 502 Gateway error with limit set to 250
Hi @mshahbaz_000 Let me check with our developers and we'll get back to you. Thanks -- Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.
Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.
Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.
-
- User Services
- Posts: 3
- Joined: Tue Feb 14, 2023 3:48 pm America/New_York
Re: Request's response 502 Gateway error with limit set to 250
@mshahbaz_000,
There are no limitations for the number of granules to request through the STAC catalog however response times can differ between datasets. The 502 error is due to a timing out issue because the requests are taking a long time to download and the network on the backend is closing. We recommend breaking down the requests into smaller segments that way the downloading is quicker.
If interested in, we have a tutorial on utilizing the CMR-STAC API https://git.earthdata.nasa.gov/projects/LPDUR/repos/data-discovery---cmr-stac-api-in-python/browse/notebooks/Data%20Discovery%20-%20CMR-STAC%20API.ipynb
If you continue to have issues, please send us your entire script and we can try to replicate the problem.
Thanks!
There are no limitations for the number of granules to request through the STAC catalog however response times can differ between datasets. The 502 error is due to a timing out issue because the requests are taking a long time to download and the network on the backend is closing. We recommend breaking down the requests into smaller segments that way the downloading is quicker.
If interested in, we have a tutorial on utilizing the CMR-STAC API https://git.earthdata.nasa.gov/projects/LPDUR/repos/data-discovery---cmr-stac-api-in-python/browse/notebooks/Data%20Discovery%20-%20CMR-STAC%20API.ipynb
If you continue to have issues, please send us your entire script and we can try to replicate the problem.
Thanks!
-
- Posts: 3
- Joined: Thu Nov 09, 2023 7:11 pm America/New_York
Re: Request's response 502 Gateway error with limit set to 250
Thank you very much for your response. I will then make sure the requests are broken to small segments.