Rate limit API?

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
jorisvandek
Posts: 2
Joined: Mon Dec 05, 2022 5:36 am America/New_York
Answers: 0

Rate limit API?

by jorisvandek » Mon Dec 05, 2022 5:48 am America/New_York

Hi all,

When downloading larger numbers of files for VIIRS we are receiving 'HTTP 429 too many requests' errors. What are the rate limits or throttling behaviors of the VIIRS APIs?

We are seeing these errors:
ERROR NUMBER 1

2022-12-05 11:20:30 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - 17 - LOGS_CAPTURED - Started capturing logs in process (pid: 17).
2022-12-05 11:20:30 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - 17 - search_noaa20_viirs_l1 - STEP_START - Started execution of step "search_noaa20_viirs_l1".
2022-12-05 11:20:32 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - search_noaa20_viirs_l1 - url: https://ladsweb.modaps.eosdis.nasa.gov/api/v1/files/product=VJ102IMG&collection=5201&dateRanges=2021-04-25..2021-04-25&areaOfInterest=x-180.0000000000y60.0000000000000,x180.0000000000y-60.0000000000000&dayCoverage=true&dnboundCoverage=false
2022-12-05 11:20:32 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - search_noaa20_viirs_l1 - 429
2022-12-05 11:20:32 +0100 - dagster - ERROR - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - 17 - search_noaa20_viirs_l1 - STEP_FAILURE - Execution of step "search_noaa20_viirs_l1" failed.
dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "search_noaa20_viirs_l1"::
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Stack Trace:
File "/dagster_env/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
yield
File "/dagster_env/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 447, in iterate_with_context
next_output = next(iterator)
File "/opt/dagster/app/projects/sourcing/viirs/job_noaa20_viirs_l1_v2.py", line 122, in search_noaa20_viirs_l1
datasets_to_produce = define_datasets_to_produce(
File "/opt/dagster/app/projects/sourcing/viirs/discover.py", line 48, in define_datasets_to_produce
data_sources = find_source_datasets(
File "/opt/dagster/app/projects/sourcing/viirs/discover.py", line 102, in find_source_datasets
if response.json().values():
File "/dagster_env/lib/python3.10/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

ERROR NUMBER 2:

2022-12-05 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - LOGS_CAPTURED - Started capturing logs in process (pid: 34).
2022-12-05 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - STEP_START - Started execution of step "process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319]".
2022-12-05 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - Loading file from: /opt/dagster/dagster_home/storage/a8c8df35-e675-44c4-bb32-f8231f471916/search_noaa20_viirs_l1/result/84bc3b4f6c664ae8a2810c79b1241319
2022-12-05 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - LOADED_INPUT - Loaded input "dataset" using input manager "io_manager", from output "result" of step "search_noaa20_viirs_l1"
2022-12-05 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - STEP_INPUT - Got input "dataset" of type "Any". (Type check passed).
2022-12-05 11:29:00 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - processing job: {'source_id': UUID('7a69fdfa-3ce2-4b64-9be1-a804edbd4c93'), 'geometry': 'POLYGON ((57.3301887512207 14.0692758560181, 85.2637481689453 18.2861347198486, 89.4081497192383 -2.41183042526245, 62.4217872619629 -6.49754762649536, 57.3301887512207 14.0692758560181))', 'date': datetime.datetime(2021, 1, 6, 8, 24), 'data_filename': 'VJ102IMG.A2021006.0824.021.2021072055500.nc', 'VJ102IMG': '/archive/allData/5201/VJ102IMG/2021/006/VJ102IMG.A2021006.0824.021.2021072055500.nc', 'tiles': ['44PMQ'], 'VJ103IMG': '/archive/allData/5201/VJ103IMG/2021/006/VJ103IMG.A2021006.0824.021.2021072034831.nc', 'CLDMSK_L2_VIIRS_NOAA20': '/archive/allData/5110/CLDMSK_L2_VIIRS_NOAA20/2021/006/CLDMSK_L2_VIIRS_NOAA20.A2021006.0824.001.2021006190617.nc'}
2022-12-05 11:29:00 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - downloading https://ladsweb.modaps.eosdis.nasa.gov/api/v2/content/archives/VJ102IMG.A2021006.0824.021.2021072055500.nc
2022-12-05 11:30:17 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - downloading https://ladsweb.modaps.eosdis.nasa.gov/api/v2/content/archives/VJ103IMG.A2021006.0824.021.2021072034831.nc
HTTP GET error code: 429
HTTP GET error message: Too Many Requests
2022-12-05 11:30:17 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - downloading https://ladsweb.modaps.eosdis.nasa.gov/api/v2/content/archives/CLDMSK_L2_VIIRS_NOAA20.A2021006.0824.001.2021006190617.nc
2022-12-05 11:30:36 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - exception = [Errno -51] NetCDF: Unknown file format: b'/tmp/viirs/VJ103IMG.A2021006.0824.021.2021072034831.nc'
2022-12-05 11:30:36 +0100 - dagster - ERROR - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - STEP_FAILURE - Execution of step "process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319]" failed.
dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "process_noaa20_viirs_l1"::
OSError: [Errno -51] NetCDF: Unknown file format: b'/tmp/viirs/VJ103IMG.A2021006.0824.021.2021072034831.nc'
Stack Trace:
File "/dagster_env/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
yield
File "/dagster_env/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 447, in iterate_with_context
next_output = next(iterator)
File "/dagster_env/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 74, in _coerce_solid_compute_fn_to_iterator
result = fn(context, **kwargs) if context_arg_provided else fn(**kwargs)
File "/opt/dagster/app/projects/sourcing/viirs/job_noaa20_viirs_l1_v2.py", line 158, in process_noaa20_viirs_l1
raise e
File "/opt/dagster/app/projects/sourcing/viirs/job_noaa20_viirs_l1_v2.py", line 153, in process_noaa20_viirs_l1
results = run_viirs_process(edc, engine, dataset)
File "/opt/dagster/app/projects/sourcing/viirs/swath2tile.py", line 84, in run_viirs_process
result = project_to_tiles(
File "/opt/dagster/app/projects/sourcing/viirs/process.py", line 181, in project_to_tiles
i05_tiled = swath2tile(data_path, geo_path,
File "/opt/dagster/app/projects/sourcing/viirs/process.py", line 113, in swath2tile
geo_file_in = nc_Dataset(geolocation_product, )
File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success

Tags:

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Rate limit API?

by LAADS_UserServices_M » Mon Dec 05, 2022 2:11 pm America/New_York

Each load balanced server has a request limit of 25 requests per second per IP. We have 4 load balanced servers so that means if perfectly load balanced we have a request limit of 100 requests/second per IP.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

jorisvandek
Posts: 2
Joined: Mon Dec 05, 2022 5:36 am America/New_York
Answers: 0

Re: Rate limit API?

by jorisvandek » Mon Dec 12, 2022 4:13 am America/New_York

Thanks for your quick response.
A follow-up question: Does this mean we can start 100 requests per second or is that 100 concurrent running requests (while files are downloading)

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Rate limit API?

by LAADS_UserServices_M » Mon Dec 12, 2022 10:16 am America/New_York

Yes, that's 100 concurrent running requests.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

Post Reply