file_search slow for larger queries

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
benraymond
Posts: 3
Joined: Thu Apr 30, 2020 7:48 am America/New_York
Answers: 0

file_search slow for larger queries

by benraymond » Thu Apr 30, 2020 7:39 pm America/New_York

The file_search API seems very slow with larger queries, should I expect this? For example, searching for "S*L3m_MO_CHL_chlor_a_9km.nc" (all monthly 9km L3m SeaWiFS chl) yesterday took 96s and returns 150ish file matches. In contrast, the same query conducted year-by-year, looping over all years 1997-2010 took 11.2s in total.

Is this expected? I note that the file search page suggests that queries of the form "S200312[0-9]*MLAC" may "take a while to process", but it's not clear what "form" is being referred to there. Is it the regexp digit match that is slow? Or the mid-string glob? In either case, the mid-string glob in my SeaWiFS example doesn't seem to be problematic when only a single year is being searched at a time, so I don't think this is the root issue.

Thanks for any pointers.

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

file_search slow for larger queries

by OB.DAAC - SeanBailey » Fri May 01, 2020 9:57 am America/New_York

Query times can vary for a number of reasons, some of which are out of our control.  In your case however, you can do something to get a better response.  Be more specific.

Here's you original query:

$ curl -d "addurl=1&results_as_file=1&search=S*L3m_MO_CHL_chlor_a_9km.nc" https://oceandata.sci.gsfc.nasa.gov/api/file_search -o test.lst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14190  100 14129    0    61    131      0  0:01:47  0:01:47 --:--:--  2877


Yep, kinda slow.  But likely you'll only ever need to do that one once, so not too bad in the grand scheme of things :razz:

But what if you limited the search to only Level 3 mapped data?
$ curl -d "dtype=L3m&addurl=1&results_as_file=1&search=S*L3m_MO_CHL_chlor_a_9km.nc" https://oceandata.sci.gsfc.nasa.gov/api/file_search -o test.lst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14200  100 14129  100    71   6040     30  0:00:02  0:00:02 --:--:--  6068


Hey! 2 seconds, not too shabby!

But what if we go crazy and limit it to only SeaWiFS L3 mapped data:
$ curl -d "sensor=seawifs&dtype=L3m&addurl=1&results_as_file=1&search=S*L3m_MO_CHL_chlor_a_9km.nc" https://oceandata.sci.gsfc.nasa.gov/api/file_search -o test.lst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14215  100 14129  100    86  30125    183 --:--:-- --:--:-- --:--:-- 30309


If you blinked, you missed it...

Sean

dana.r.wilson
Posts: 71
Joined: Mon Apr 20, 2020 8:04 am America/New_York
Answers: 0

file_search slow for larger queries

by dana.r.wilson » Fri May 01, 2020 10:08 am America/New_York

benraymond
We are looking into the issue, a few pointers would be choosing a specific mission and or type in your case selecting the level3 smi option from the web interface would help or if you are using command line dtype=L3m and sensor=Seawifs.

benraymond
Posts: 3
Joined: Thu Apr 30, 2020 7:48 am America/New_York
Answers: 0

file_search slow for larger queries

by benraymond » Fri May 01, 2020 6:13 pm America/New_York

Perfecto. Thanks!!

benraymond
Posts: 3
Joined: Thu Apr 30, 2020 7:48 am America/New_York
Answers: 0

file_search slow for larger queries

by benraymond » Fri May 01, 2020 6:16 pm America/New_York

@dana - no need to look further on my behalf, specifying the extra details is perfectly feasible and looks like it will solve my issue. Ta.

Post Reply