File Search broken ?

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: File Search broken ?

by OB.DAAC - SeanBailey » Tue Jun 28, 2022 1:19 pm America/New_York

We are investigating, but the problem is not consistent so diagnosing it is proving a challenge. Stay tuned.

Sean

Tags:

dem1
Posts: 82
Joined: Mon Nov 28, 2005 4:49 am America/New_York
Answers: 0

Re: File Search broken ?

by dem1 » Thu Jun 30, 2022 6:45 am America/New_York

Thanks a lot Sean!
I have done more analysis:
  • previously I split my search requests by sensor and month
  • I tried to reduce the split period to 10 days but I still had timeouts, see the detailed log below: most of the time the requests are fast (note that our code ensure a minimum delay of 5s between each HTTP request to your API), but some cases are much more longer (25s then 60s in the log), seems like a lag in your API?
  • then I tried to reduce the split period to 5 days, it seems working better with an homogeneous requests duration and I haven't any timeout, so for the moment I will use this workaround
Log when splitting searchs by 10 days:

Code: Select all

2022-06-30 10:52:04,327 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-02-16', 'edate': '2022-02-26', 'psdate': '2021-12-12', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-06-30 10:52:09,331 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-02-27', 'edate': '2022-03-09', 'psdate': '2021-12-12', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-06-30 10:52:14,334 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-03-10', 'edate': '2022-03-20', 'psdate': '2021-12-12', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-06-30 10:52:39,912 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-03-21', 'edate': '2022-03-31', 'psdate': '2021-12-12', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-06-30 10:53:39,198 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-04-01', 'edate': '2022-04-11', 'psdate': '2021-12-12', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-06-30 10:53:44,202 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-04-12', 'edate': '2022-04-22', 'psdate': '2021-12-12', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
...
2022-06-30 10:55:54,319 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'aqua', 'sdate': '2022-02-16', 'edate': '2022-02-26', 'psdate': '2021-12-12', 'pedate': None, 'dtype': 'L2', 'search': 'A*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-06-30 10:55:59,323 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'aqua', 'sdate': '2022-02-27', 'edate': '2022-03-09', 'psdate': '2021-12-12', 'pedate': None, 'dtype': 'L2', 'search': 'A*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-06-30 10:56:04,327 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'aqua', 'sdate': '2022-03-10', 'edate': '2022-03-20', 'psdate': '2021-12-12', 'pedate': None, 'dtype': 'L2', 'search': 'A*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-06-30 11:01:04,430 [ERROR] requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=300)

dem1
Posts: 82
Joined: Mon Nov 28, 2005 4:49 am America/New_York
Answers: 0

Re: File Search broken ?

by dem1 » Wed Jul 06, 2022 7:47 am America/New_York

Hi,

The issue seems getting worse: since 2022-07-05 4pm UTC my workaround to split by 5 days no more works and at least this request always timeouts (after 300s):

Code: Select all

2022-07-06 08:38:49,753 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'aqua', 'sdate': '2021-12-18', 'edate': '2021-12-23', 'psdate': '2021-12-18', 'pedate': None, 'dtype': 'L2', 'search': 'A*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
I also see that the other successful requests are longer than usual, often more than 20 seconds, example:

Code: Select all

2022-07-06 08:06:01,766 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2021-12-30', 'edate': '2022-01-04', 'psdate': '2021-12-18', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-07-06 08:06:27,798 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-01-05', 'edate': '2022-01-10', 'psdate': '2021-12-18', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-07-06 08:06:53,273 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-01-11', 'edate': '2022-01-16', 'psdate': '2021-12-18', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-07-06 08:07:18,678 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-01-17', 'edate': '2022-01-22', 'psdate': '2021-12-18', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
I tried to reduce the search period to only two days then one day, it was better but I still get timeouts! Example:

Code: Select all

2022-07-06 11:05:21,209 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'aqua', 'sdate': '2021-12-18', 'edate': '2021-12-19', 'psdate': '2021-12-18', 'pedate': None, 'dtype': 'L2', 'search': 'A*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
I don't know if it helps but the timeouts seem happen for Aqua, where Terra requests have no issues.
For the moment I was able to avoid timeouts by using one day and search products only for the past 20 days.

Julien

dem1
Posts: 82
Joined: Mon Nov 28, 2005 4:49 am America/New_York
Answers: 0

Re: File Search broken ?

by dem1 » Mon Sep 05, 2022 3:46 am America/New_York

Hi,

Since 3 weeks we successfully got back to search requests by 30-days periods, but the same problem came back since 01 September 4:00pm UTC, the searchs timeout after 5 minutes and you can see on the log of this morning below that some requests are successful but are very long, more than 3-4 minutes. Logs just before the problem came back show that everything was OK with requests duration around 5 seconds, so it seems something get wrong since the 01 September 4:00pm UTC. For the moment I switch back to requests by 5-days periods and it seems work (but I also had the issue with 5-days in the past).

Code: Select all

2022-09-05 08:35:02,584 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-02-17', 'edate': '2022-03-18', 'psdate': '2022-08-06', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-09-05 08:38:58,341 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-03-19', 'edate': '2022-04-17', 'psdate': '2022-08-06', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-09-05 08:42:28,889 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-04-18', 'edate': '2022-05-17', 'psdate': '2022-08-06', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-09-05 08:45:38,449 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-05-18', 'edate': '2022-06-16', 'psdate': '2022-08-06', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-09-05 08:49:03,512 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-06-17', 'edate': '2022-07-16', 'psdate': '2022-08-06', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-09-05 08:52:25,832 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-07-17', 'edate': '2022-08-15', 'psdate': '2022-08-06', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-09-05 08:55:47,993 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2022-08-16', 'edate': '2022-09-05', 'psdate': '2022-08-06', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-09-05 08:59:09,985 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'aqua', 'sdate': '2022-02-17', 'edate': '2022-03-18', 'psdate': '2022-08-06', 'pedate': None, 'dtype': 'L2', 'search': 'AQUA_MODIS.*.L2.OC*.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
2022-09-05 09:04:10,241 requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=300)
Thanks,
Julien

Post Reply