Direct Data Access, how to retrieve metadata?

[starbright]

I'm wondering if there is a pythonic way to retrieve metadata of netcdf4 files stored at the "Direct Data Access", such as sensing time and longitude/latitude bounds? I need to look at those information before making the decision whether to start a download.For example, this file:https://oceandata.sci.gsfc.nasa.gov (/cgi/getfile/A2017001000000.L2_LAC_OC.nc) which is listed at this location:https://oceandata.sci.gsfc.nasa.gov (/MODIS-Aqua/L2/2017/001/). I'm really new to Python and have Googled for two days but could not figure out how. Is it even possible with the HTTPS server?Very much appreciated!

[OB WebDev - norman]

Hi,Our level-2 browser is designed to perform geographical andtemporal searches of the data in our archive.https://oceancolor.gsfc.nasa.gov (/cgi/browse.pl?file=A2017001000000.L2_LAC_OC). Please click on the Help button on that page for more information.Regards,Norman

[OB.DAAC - SeanBailey]

Using the level-2 browser you can also request data to be extracted to your region of interest (assuming your region is smaller than a typical 5-min granule).Additionally, if you need to acquire data for the future you can set up a data subscription (either extracted or non-extracted).If you're really into command-line processing, you can directly access the ESDIS Common Metadata Repository:which backends the Earthdata search (https://search.earthdata.nasa.gov/search)(and which we populate with our file metadata).  Here's an example for finding the MODIS-Aqua Level-2 ocean color files that match the Gulf of Mexico:curl -i "https://cmr.earthdata.nasa.gov/search/granules.json?provider=OB_DAAC&short_name=MODISA_L2_OC&bounding_box=-105,18,-82,30&page_size=10&page_num=1"(the output JSON file is attached)In this example, I've used the default the page_size of 10 granules - the max the CMR API allows is 2000. If the query would return more granules than the requested page_size  , you'd have to run it again with an increment to the page_num option until you get the full set.The header information (returned by the -i option to curl) includes a custom field (CMR-Hits) with the total matched granules:HTTP/1.1 200 OKAccess-Control-Allow-Origin: *Access-Control-Expose-Headers: CMR-Hits, CMR-Request-IdCMR-Hits: 14706CMR-Request-Id: 8d059213-61f1-4d7c-bec3-9a698238f655CMR-Took: 106Content-Type: text/csv; charset=utf-8Date: Thu, 08 Jun 2017 15:06:25 GMTServer: nginxStrict-Transport-Security: max-age=31536000; includeSubDomains;Content-Length: 2008Connection: keep-aliveRegards,Seanattachment 1

[starbright]

Awesome! That's exactly what I was looking for! Thanks, Sean and Norman.

[starbright]

Sean,

One problem is that there seems no near real time data. For example, if I put this link in, it returns essentially empty results.

https://cmr.earthdata.nasa.gov/search/granules.json?provider=OB_DAAC&short_name=MODISA_L2_OC&page_size=100&page_num=1&polygon=10,10,30,10,30,20,10,20,10,10&temporal=2017-06-08T00:00:00Z,%202017-06-08T23:59:59Z

How could that be?

[OB.DAAC - SeanBailey]

That's because we don't put the data into CMR until we've processed the file as "refined" - which can be up to a two-week delay.
If you want NRT, create a subscription.

Sean

[starbright]

In the NRT data subscription, is there a way to specify a polygon as opposed to a bounding box?

[OB.DAAC - SeanBailey]

No.  The information available for the search is rectilinear, so we stick to boxes on a sphere...

[starbright]

I'm trying to think of a way to work around this limitation. If I create multiple boxes that partly overlap one another, how will your server handle the multiple boxes? Will your FTP server put a bunch of data granules pooled in one folder, or multiple folders (each corresponding to one box)? If the latter will be the case, I'm afraid there might be redundancies of data granules.

I could not do test subscriptions to answer the questions myself because I created a test subscription yesterday (with FTP as the option to receive data), but got no emails so far notifying me where to get the data.

I know it's a lot of questions. Thank you so much for the patience!

[OB.DAAC - SeanBailey]

Yes, there will be redundant granules as each subscription is treated separately.
Just make a single request with a box big enough to cover your ROI polygon.

Sean