Request's response 502 Gateway error with limit set to 250

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
mshahbaz_000
Posts: 3
Joined: Thu Nov 09, 2023 7:11 pm America/New_York
Answers: 0

Request's response 502 Gateway error with limit set to 250

by mshahbaz_000 » Thu Nov 09, 2023 7:31 pm America/New_York

Through a python script, I am sending a POST request to retrieve items from LPCLOUD STAC Catalog. My request parameters are as follows:

params = {'limit': 250, 'bbox': '-112.63573388436109,56.74522108547581,-110.83605841365043,57.74219208744419', 'datetime': '2023-01-02T00:00:00Z/2023-11-09T23:59:59Z', 'collections': ['HLSS30.v2.0']}

The POST command is like this:
items = requests.post(lp_search, json=params)

with lp_search being the search endpoint for the LPCLOUD STAC Catalog:
https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search

This results in response status code of 502 (bad Gateway server error). The way I can fix it is by reducing the 'limit' to 100.

I am wondering why the error happens with larger limits? In some requests (e.g. with a much shorter datetime range and a much smaller bbox), the limit of 250 matched items works fine. But, in some requests like the one above, a smaller limit should be used. Is there a safe way to determine this limit so that we would not get an unexpected response error?

Tags:

LP DAAC - dgolon
User Services
User Services
Posts: 276
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 15 times
Been thanked: 2 times
Contact:

Re: Request's response 502 Gateway error with limit set to 250

by LP DAAC - dgolon » Mon Nov 13, 2023 9:27 am America/New_York

Hi @mshahbaz_000 Let me check with our developers and we'll get back to you. Thanks -- Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

LP DAAC - kyrad
User Services
User Services
Posts: 3
Joined: Tue Feb 14, 2023 3:48 pm America/New_York
Answers: 0

Re: Request's response 502 Gateway error with limit set to 250

by LP DAAC - kyrad » Tue Nov 14, 2023 9:04 am America/New_York

@mshahbaz_000,
There are no limitations for the number of granules to request through the STAC catalog however response times can differ between datasets. The 502 error is due to a timing out issue because the requests are taking a long time to download and the network on the backend is closing. We recommend breaking down the requests into smaller segments that way the downloading is quicker. 

If interested in, we have a tutorial on utilizing the CMR-STAC API https://git.earthdata.nasa.gov/projects/LPDUR/repos/data-discovery---cmr-stac-api-in-python/browse/notebooks/Data%20Discovery%20-%20CMR-STAC%20API.ipynb 

If you continue to have issues, please send us your entire script and we can try to replicate the problem.
Thanks!

mshahbaz_000
Posts: 3
Joined: Thu Nov 09, 2023 7:11 pm America/New_York
Answers: 0

Re: Request's response 502 Gateway error with limit set to 250

by mshahbaz_000 » Tue Nov 14, 2023 5:34 pm America/New_York

Thank you very much for your response. I will then make sure the requests are broken to small segments.

Post Reply