Quick listing of all files for a product?

Please enter here to ask a question about any NASA Science related topics!
Post Reply
simonf
Posts: 4
Joined: Wed Nov 21, 2018 9:36 am America/New_York

Quick listing of all files for a product?

by simonf » Thu Nov 29, 2018 4:36 pm America/New_York

Hi,

Would it be possible to either create a listing of all files and their modification times that can be retrieved in one fetch per dataset?

I am the data catalog maintainer for Google Earth Engine. We fully mirror several ocean color datasets, and we would like to reingest older assets if their files get modified, but it's hard to detect that. The only way to do this is to rescan all of the directories for each product daily, which is brittle and time-consuming.

A solution for this would be to create a single per-product listing that stores all of this metadata (for example, LP DAAC does this for Landsat data).

We are working with NASA EOSDIS to try and make some progress on this issue in general, but meanwhile some temporary solution would be much appreciated.

Thanks,
Simon

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1195
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

Quick listing of all files for a product?

by OB.DAAC - SeanBailey » Thu Nov 29, 2018 7:59 pm America/New_York

Simon,

You can use the file_search api to retrieve such a listing.  Using your POC daily mapped files as an example:

wget --post-data="search=A20*L3m_DAY_POC_poc_4km.nc&dtype=L3m&sensor=aqua&format=json&std_only=1" https://oceandata.sci.gsfc.nasa.gov/api/file_search -O poc-daily.json


Sean
attachment 1

simonf
Posts: 4
Joined: Wed Nov 21, 2018 9:36 am America/New_York

Quick listing of all files for a product?

by simonf » Fri Nov 30, 2018 1:41 pm America/New_York

Thank you! I tried using search=A2*.nc to get all the Aqua file listings, but this request times out. So I have to request each variable individually. (This is why I prefer files with static listings over API calls, as APIs usually cannot handle large results.)

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1195
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

Quick listing of all files for a product?

by OB.DAAC - SeanBailey » Fri Nov 30, 2018 3:01 pm America/New_York

Simon,

Yes, dumping the entire archive might take longer than the timeout window for the API...but I hate static files as I have to make sure they're updated regularly...the API will always be current.

Since the vast majority of the data are not going to change between reprocessing events (which we announce and you should subscribe to our mailing list to be made aware of these),
you can add the option psdate=<last-date-run>, e.g.

wget --post-data="search=A20*nc&dtype=L3m&sensor=aqua&format=json&std_only=1&psdate=2018-10-01" https://oceandata.sci.gsfc.nasa.gov/api/file_search -O l3-daily.json

The 'p' refers to the processed date.  The 's' (as in ps) refers to the start range - an equivalent end range also exists.  So you can search on a range of dates.See the FAQ on the file_search utility.

Sean

moeinizadi
Posts: 11
Joined: Mon Sep 09, 2019 9:44 am America/New_York

Quick listing of all files for a product?

by moeinizadi » Thu Jul 02, 2020 12:03 pm America/New_York

I am downloading MODIS level-03 ocean color parameters such as Chl, Sea Surface Temperature as time-series format from google earth engine. I have thousands of observations and I don't know how to adjust my code to download all the points at once. Is there anyone so I can share my google earth engine code and take a look at it?

Thank you,
Mo.

Post Reply