File Search broken ?

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
dem1
Posts: 67
Joined: Mon Nov 28, 2005 4:49 am America/New_York

File Search broken ?

by dem1 » Tue Jun 14, 2022 4:09 am America/New_York

Hi,

Since yesterday 2022-06-13 04:00:00PM UTC the file search API https://oceandata.sci.gsfc.nasa.gov/api/file_search/ seems broken depending on search parameters:
for example when selecting Start=2022-05-06 End=2022-05-20 DataType=Level2 Mission=MODIS Aqua the search never ends, while with Start=2022-05-15 it succeed quickly. Note that the issue seems varying because 5 minutes ago Start=2022-05-07 was also quick, but now it blocks too.

Thanks,
Julien

Tags:

OB WebDev - dana.r.wilson
Subject Matter Expert
Subject Matter Expert
Posts: 66
Joined: Mon Apr 20, 2020 8:04 am America/New_York

Re: File Search broken ?

by OB WebDev - dana.r.wilson » Tue Jun 14, 2022 6:36 am America/New_York

We are looking into the issue.

mcoyne
Posts: 7
Joined: Wed Aug 31, 2016 5:41 pm America/New_York

Re: File Search broken ?

by mcoyne » Tue Jun 14, 2022 2:14 pm America/New_York

If it helps, it seems to consistently break at result 1680 for me (or 131072 bytes). I haven't tested it, but it may work correctly for smaller searches. Each time I try a search I get the same error:

"Connection closed at byte 131072"

OB WebDev - dana.r.wilson
Subject Matter Expert
Subject Matter Expert
Posts: 66
Joined: Mon Apr 20, 2020 8:04 am America/New_York

Re: File Search broken ?

by OB WebDev - dana.r.wilson » Tue Jun 14, 2022 2:20 pm America/New_York

The issue has been resolved.

mcoyne
Posts: 7
Joined: Wed Aug 31, 2016 5:41 pm America/New_York

Re: File Search broken ?

by mcoyne » Tue Jun 14, 2022 3:02 pm America/New_York

File search is still breaking for me after receiving 131072 bytes.

OB.DAAC - amscott
User Services
User Services
Posts: 219
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 2 times

Re: File Search broken ?

by OB.DAAC - amscott » Tue Jun 14, 2022 3:09 pm America/New_York

Hi,

Would you mind walking us through your steps so we can try to recreate your issue? Are you using the web interface or the command line implementation? When I try the web interface using any of the dates and information you supplied, I receive expected data. If command line, please share your command.

mcoyne
Posts: 7
Joined: Wed Aug 31, 2016 5:41 pm America/New_York

Re: File Search broken ?

by mcoyne » Tue Jun 14, 2022 3:29 pm America/New_York

Sure. I am using the command line implementation. Here's an example of code that was working last Friday, but does not work today:

wget --post-data="dtype=L3m&addurl=1&results_as_file=1&search=*DAY_*CHL_chlor_a_4km.nc&cksum=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

Sorry, that should be all one line, but I can't seem to get BBCode to turn "on" to format the code.

OB WebDev - dana.r.wilson
Subject Matter Expert
Subject Matter Expert
Posts: 66
Joined: Mon Apr 20, 2020 8:04 am America/New_York

Re: File Search broken ?

by OB WebDev - dana.r.wilson » Fri Jun 17, 2022 4:14 pm America/New_York

The issue has been resolved.

dem1
Posts: 67
Joined: Mon Nov 28, 2005 4:49 am America/New_York

Re: File Search broken ?

by dem1 » Mon Jun 27, 2022 5:12 am America/New_York

Hi,

The same problem came back since 25 June 10:00pm UTC, the search timeouts after 5 minutes depending on period (and the issue seems varying from one request to another).

Thanks,
Julien

dem1
Posts: 67
Joined: Mon Nov 28, 2005 4:49 am America/New_York

Re: File Search broken ?

by dem1 » Mon Jun 27, 2022 6:01 am America/New_York

Example of request which always fails:

Code: Select all

2022-06-27 11:41:32,495 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'aqua', 'sdate': '2021-12-09', 'edate': '2021-12-31', 'psdate': '2021-12-09', 'pedate': None, 'dtype': 'L2', 'search': 'A*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=300)
I tried to reproduce the timeout on the same request using the file search API webpage but I have no timeout (last time I were able to reproduce it).

On the other side the same request for Terra works well without timeout:

Code: Select all

2022-06-27 11:40:02,771 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2021-12-09', 'edate': '2021-12-31', 'psdate': '2021-12-09', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})

Post Reply