Error 429 VIIRS acquisition

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
alexis_fontana
Posts: 1
Joined: Wed Mar 15, 2023 1:50 am America/New_York
Answers: 0

Error 429 VIIRS acquisition

by alexis_fontana » Wed Mar 15, 2023 2:06 am America/New_York

Hello,

I apologize if this is the wrong place to post this request, but I'm stuck with an error I don't know how to interpret. We're downloading files for VIIRS and I can't find documentation about what code 429 means in this case? It used to work before but I don't know what changed. We hadn't run this code in about 6 months.

Here's what's returned:

2023-03-14 11:20:30 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - 17 - LOGS_CAPTURED - Started capturing logs in process (pid: 17).
2023-03-14 11:20:30 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - 17 - search_noaa20_viirs_l1 - STEP_START - Started execution of step "search_noaa20_viirs_l1".
2023-03-14 11:20:32 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - search_noaa20_viirs_l1 - url: https://ladsweb.modaps.eosdis.nasa.gov/api/v1/files/product=VJ102IMG&collection=5201&dateRanges=2021-04-25..2021-04-25&areaOfInterest=x-180.0000000000y60.0000000000000,x180.0000000000y-60.0000000000000&dayCoverage=true&dnboundCoverage=false
2023-03-14 11:20:32 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - search_noaa20_viirs_l1 - 429
2023-03-14 11:20:32 +0100 - dagster - ERROR - noaa20_viirs_l1_v2 - 5f1b1fb9-3809-4976-94ca-a980ececed99 - 17 - search_noaa20_viirs_l1 - STEP_FAILURE - Execution of step "search_noaa20_viirs_l1" failed.
dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "search_noaa20_viirs_l1"::
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Stack Trace:
File "/dagster_env/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
yield
File "/dagster_env/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 447, in iterate_with_context
next_output = next(iterator)
File "/opt/dagster/app/projects/sourcing/viirs/job_noaa20_viirs_l1_v2.py", line 122, in search_noaa20_viirs_l1
datasets_to_produce = define_datasets_to_produce(
File "/opt/dagster/app/projects/sourcing/viirs/discover.py", line 48, in define_datasets_to_produce
data_sources = find_source_datasets(
File "/opt/dagster/app/projects/sourcing/viirs/discover.py", line 102, in find_source_datasets
if response.json().values():
File "/dagster_env/lib/python3.10/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

2023-03-14 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - LOGS_CAPTURED - Started capturing logs in process (pid: 34).
2023-03-14 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - STEP_START - Started execution of step "process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319]".
2023-03-14 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - Loading file from: /opt/dagster/dagster_home/storage/a8c8df35-e675-44c4-bb32-f8231f471916/search_noaa20_viirs_l1/result/84bc3b4f6c664ae8a2810c79b1241319
2023-03-14 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - LOADED_INPUT - Loaded input "dataset" using input manager "io_manager", from output "result" of step "search_noaa20_viirs_l1"
2023-03-14 11:29:00 +0100 - dagster - DEBUG - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - STEP_INPUT - Got input "dataset" of type "Any". (Type check passed).
2023-03-14 11:29:00 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - processing job: {'source_id': UUID('7a69fdfa-3ce2-4b64-9be1-a804edbd4c93'), 'geometry': 'POLYGON ((57.3301887512207 14.0692758560181, 85.2637481689453 18.2861347198486, 89.4081497192383 -2.41183042526245, 62.4217872619629 -6.49754762649536, 57.3301887512207 14.0692758560181))', 'date': datetime.datetime(2021, 1, 6, 8, 24), 'data_filename': 'VJ102IMG.A2021006.0824.021.2021072055500.nc', 'VJ102IMG': '/archive/allData/5201/VJ102IMG/2021/006/VJ102IMG.A2021006.0824.021.2021072055500.nc', 'tiles': ['44PMQ'], 'VJ103IMG': '/archive/allData/5201/VJ103IMG/2021/006/VJ103IMG.A2021006.0824.021.2021072034831.nc', 'CLDMSK_L2_VIIRS_NOAA20': '/archive/allData/5110/CLDMSK_L2_VIIRS_NOAA20/2021/006/CLDMSK_L2_VIIRS_NOAA20.A2021006.0824.001.2021006190617.nc'}
2023-03-14 11:29:00 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - https://bestcanadianflorists.com process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - downloading https://ladsweb.modaps.eosdis.nasa.gov/api/v2/content/archives/VJ102IMG.A2021006.0824.021.2021072055500.nc
2023-03-14 11:30:17 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - downloading https://ladsweb.modaps.eosdis.nasa.gov/api/v2/content/archives/VJ103IMG.A2021006.0824.021.2021072034831.nc
HTTP GET error code: 429
HTTP GET error message: Too Many Requests
2023-03-14 11:30:17 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - downloading https://ladsweb.modaps.eosdis.nasa.gov/api/v2/content/archives/CLDMSK_L2_VIIRS_NOAA20.A2021006.0824.001.2021006190617.nc
2023-03-14 11:30:36 +0100 - dagster - INFO - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - exception = [Errno -51] NetCDF: Unknown file format: b'/tmp/viirs/VJ103IMG.A2021006.0824.021.2021072034831.nc'
2023-03-14 11:30:36 +0100 - dagster - ERROR - noaa20_viirs_l1_v2 - a8c8df35-e675-44c4-bb32-f8231f471916 - 34 - process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319] - STEP_FAILURE - Execution of step "process_noaa20_viirs_l1[84bc3b4f6c664ae8a2810c79b1241319]" failed.
dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "process_noaa20_viirs_l1"::
OSError: [Errno -51] NetCDF: Unknown file format: b'/tmp/viirs/VJ103IMG.A2021006.0824.021.2021072034831.nc'
Stack Trace:
File "/dagster_env/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
yield
File "/dagster_env/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 447, in iterate_with_context
next_output = next(iterator)
File "/dagster_env/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 74, in _coerce_solid_compute_fn_to_iterator
result = fn(context, **kwargs) if context_arg_provided else fn(**kwargs)
File "/opt/dagster/app/projects/sourcing/viirs/job_noaa20_viirs_l1_v2.py", line 158, in process_noaa20_viirs_l1
raise e
File "/opt/dagster/app/projects/sourcing/viirs/job_noaa20_viirs_l1_v2.py", line 153, in process_noaa20_viirs_l1
results = run_viirs_process(edc, engine, dataset)
File "/opt/dagster/app/projects/sourcing/viirs/swath2tile.py", line 84, in run_viirs_process
result = project_to_tiles(
File "/opt/dagster/app/projects/sourcing/viirs/process.py", line 181, in project_to_tiles
i05_tiled = swath2tile(data_path, geo_path,
File "/opt/dagster/app/projects/sourcing/viirs/process.py", line 113, in swath2tile
geo_file_in = nc_Dataset(geolocation_product, )
File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success

Thank you for any information you may have.

Best regards,
Alexis

Tags:

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Error 429 VIIRS acquisition

by LAADS_UserServices_M » Wed Mar 15, 2023 12:56 pm America/New_York

Dear Alexis:
Just ran your command and got result without any issue (see attached). I could something temporary so please try again.
screen_shot.png
screen_shot.png (64.87 KiB) Not viewed yet
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

Post Reply