Search with wget won't work

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
oceanwatchcp
Posts: 8
Joined: Mon Oct 24, 2022 4:51 pm America/New_York
Answers: 0

Search with wget won't work

by oceanwatchcp » Mon Oct 24, 2022 8:38 pm America/New_York

Tried the example for command line interface provided on the "Search and Download" page:

wget -q --post-data="sensor=octs&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

Got an error message:
<h1>An error has occurred</h1>
Filename pattern invalid

Can I get help on how to modify the search? Thank you!

Tags:

OB.DAAC - amscott
User Services
User Services
Posts: 350
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Search with wget won't work

by OB.DAAC - amscott » Tue Oct 25, 2022 8:57 am America/New_York

Try: "O*DAY_CHL*"

wget -q --post-data="results_as_file=1&std_only1&subType1&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&search=O*DAY_CHL*&sensor_id=9" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

oceanwatchcp
Posts: 8
Joined: Mon Oct 24, 2022 4:51 pm America/New_York
Answers: 0

Re: Search with wget won't work

by oceanwatchcp » Tue Oct 25, 2022 5:50 pm America/New_York

Thanks! This works!

How should I modify the search command to be able make it work in general? My goal is to get a list of files for daily and monthly MODIS Aqua chlorophyll-a data files, and check for new files to download (so that the data is up to date locally).

I tried to search the MODIS Aqua chlorophyll-a data, this is the command generated on the page after a manual search on the API. But it didn't work on my computer.

wget -q --post-data="results_as_file=1&std_only1&subType1&search=A2019*.L3m_MO_CHL_chlor_a_4km.nc" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

Would you help me figure out how to modify this one and alike? Thank you very much!
Daisy

OB.DAAC - amscott
User Services
User Services
Posts: 350
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Search with wget won't work

by OB.DAAC - amscott » Wed Oct 26, 2022 9:25 pm America/New_York

Hi Daisy,

Your wget command works for me. In general, there are instructions here: https://oceancolor.gsfc.nasa.gov/data/download_methods/
Screen Shot 2022-10-26 at 9.19.51 PM.png
Screen Shot 2022-10-26 at 9.19.51 PM.png (141.1 KiB) Not viewed yet
We will look into what went wrong with the API search. A new version with updated search capabilities is being developed so please stay tuned.

oceanwatchcp
Posts: 8
Joined: Mon Oct 24, 2022 4:51 pm America/New_York
Answers: 0

Re: Search with wget won't work

by oceanwatchcp » Thu Oct 27, 2022 3:56 pm America/New_York

Hi, thanks for helping me!

I used a python script with the call function for the wget command:
call(['wget','-q', '--post-data="results_as_file=1&std_only1&subType1&search=A2019*.L3m_MO_CHL_chlor_a_4km.nc"','-O', '-', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search'])

But I got the below error message in the output html page:
<h1>An error has occurred</h1>
Filename pattern invalid

It is strange that the same command won't work in the python script.

The earlier search after I modified it according to your suggestion worked with the same script:
call(['wget','-q', '--post-data="results_as_file=1&std_only1&subType1&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&search=O*DAY_CHL*&sensor_id=9"','-O','-','https://oceandata.sci.gsfc.nasa.gov/api/file_search'])

Do you have an idea why and how I can fix it?

Thanks!
Daisy

OB.DAAC - amscott
User Services
User Services
Posts: 350
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Search with wget won't work

by OB.DAAC - amscott » Mon Oct 31, 2022 11:05 am America/New_York

try: call(['wget','-q', '--post-data="results_as_file=1&std_only1&subType=1&sdate=1999-01-01&edate=1999-12-31&search=A*MO_CHL_*4km*"','-O', '-', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search'])

oceanwatchcp
Posts: 8
Joined: Mon Oct 24, 2022 4:51 pm America/New_York
Answers: 0

Re: Search with wget won't work

by oceanwatchcp » Tue Nov 01, 2022 3:05 pm America/New_York

Thanks again!

I tried the command suggested, but I still got the same error message:

<h1>An error has occurred</h1>
Filename pattern invalid

I modified the search here and there, e.g., "subType1", "search=*MO_CHL*4km*"', but kept getting the same error message.

Do you have other suggestions how I can modify the commend?

Thank you,
Daisy

OB.DAAC - amscott
User Services
User Services
Posts: 350
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Search with wget won't work

by OB.DAAC - amscott » Tue Nov 01, 2022 3:19 pm America/New_York

Hi Daisy,

The search just uses regex and you can try patterns until something works. There are still some differences between your search and mine. I would recommend matching the beginning pattern of the file type you are looking for (aqua, snpp) in addition to the product suite (chl, rrs) and, if desired, resolution (4km, 9km). You can see a listing of files in the direct data access area of our site and our filenaming conventions to become more familiar with valid options to request.

Alicia

Post Reply