File Search broken ?

[dem1]

Hi,

Since yesterday 2022-06-13 04:00:00PM UTC the file search API https://oceandata.sci.gsfc.nasa.gov/api/file_search/ seems broken depending on search parameters:
for example when selecting Start=2022-05-06 End=2022-05-20 DataType=Level2 Mission=MODIS Aqua the search never ends, while with Start=2022-05-15 it succeed quickly. Note that the issue seems varying because 5 minutes ago Start=2022-05-07 was also quick, but now it blocks too.

Thanks,
Julien

[dana.r.wilson]

We are looking into the issue.

[mcoyne]

If it helps, it seems to consistently break at result 1680 for me (or 131072 bytes). I haven't tested it, but it may work correctly for smaller searches. Each time I try a search I get the same error:

"Connection closed at byte 131072"

[dana.r.wilson]

The issue has been resolved.

[mcoyne]

File search is still breaking for me after receiving 131072 bytes.

[OB.DAACx - amscott]

Hi,

Would you mind walking us through your steps so we can try to recreate your issue? Are you using the web interface or the command line implementation? When I try the web interface using any of the dates and information you supplied, I receive expected data. If command line, please share your command.

[mcoyne]

Sure. I am using the command line implementation. Here's an example of code that was working last Friday, but does not work today:

wget --post-data="dtype=L3m&addurl=1&results_as_file=1&search=*DAY_*CHL_chlor_a_4km.nc&cksum=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

Sorry, that should be all one line, but I can't seem to get BBCode to turn "on" to format the code.

[dana.r.wilson]

The issue has been resolved.

[dem1]

Hi,

The same problem came back since 25 June 10:00pm UTC, the search timeouts after 5 minutes depending on period (and the issue seems varying from one request to another).

Thanks,
Julien

[dem1]

Example of request which always fails:

Code: Select all

2022-06-27 11:41:32,495 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'aqua', 'sdate': '2021-12-09', 'edate': '2021-12-31', 'psdate': '2021-12-09', 'pedate': None, 'dtype': 'L2', 'search': 'A*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=300)

I tried to reproduce the timeout on the same request using the file search API webpage but I have no timeout (last time I were able to reproduce it).

On the other side the same request for Terra works well without timeout:

Code: Select all

2022-06-27 11:40:02,771 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2021-12-09', 'edate': '2021-12-31', 'psdate': '2021-12-09', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})