Direct Data Access, how to retrieve metadata?

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
starbright
Posts: 16
Joined: Sun Jul 31, 2005 9:10 pm America/New_York
Answers: 0

Direct Data Access, how to retrieve metadata?

by starbright » Wed Jun 07, 2017 9:14 pm America/New_York

I'm wondering if there is a pythonic way to retrieve metadata of netcdf4 files stored at the "Direct Data Access", such as sensing time and longitude/latitude bounds? I need to look at those information before making the decision whether to start a download.For example, this file:https://oceandata.sci.gsfc.nasa.gov (/cgi/getfile/A2017001000000.L2_LAC_OC.nc) which is listed at this location:https://oceandata.sci.gsfc.nasa.gov (/MODIS-Aqua/L2/2017/001/). I'm really new to Python and have Googled for two days but could not figure out how. Is it even possible with the HTTPS server?Very much appreciated!

Filters:

OB WebDev - norman
Subject Matter Expert
Subject Matter Expert
Posts: 143
Joined: Tue Feb 09, 2021 8:19 am America/New_York
Answers: 0

Direct Data Access, how to retrieve metadata?

by OB WebDev - norman » Thu Jun 08, 2017 8:29 am America/New_York

Hi,Our level-2 browser is designed to perform geographical andtemporal searches of the data in our archive.https://oceancolor.gsfc.nasa.gov (/cgi/browse.pl?file=A2017001000000.L2_LAC_OC). Please click on the Help button on that page for more information.Regards,Norman

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1485
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Direct Data Access, how to retrieve metadata?

by OB.DAAC - SeanBailey » Thu Jun 08, 2017 11:20 am America/New_York

Using the level-2 browser you can also request data to be extracted to your region of interest (assuming your region is smaller than a typical 5-min granule).Additionally, if you need to acquire data for the future you can set up a data subscription (either extracted or non-extracted).If you're really into command-line processing, you can directly access the ESDIS Common Metadata Repository:which backends the Earthdata search (https://search.earthdata.nasa.gov/search)(and which we populate with our file metadata).  Here's an example for finding the MODIS-Aqua Level-2 ocean color files that match the Gulf of Mexico:curl -i "https://cmr.earthdata.nasa.gov/search/granules.json?provider=OB_DAAC&short_name=MODISA_L2_OC&bounding_box=-105,18,-82,30&page_size=10&page_num=1"(the output JSON file is attached)In this example, I've used the default the page_size of 10 granules - the max the CMR API allows is 2000. If the query would return more granules than the requested page_size  , you'd have to run it again with an increment to the page_num option until you get the full set.The header information (returned by the -i option to curl) includes a custom field (CMR-Hits) with the total matched granules:HTTP/1.1 200 OKAccess-Control-Allow-Origin: *Access-Control-Expose-Headers: CMR-Hits, CMR-Request-IdCMR-Hits: 14706CMR-Request-Id: 8d059213-61f1-4d7c-bec3-9a698238f655CMR-Took: 106Content-Type: text/csv; charset=utf-8Date: Thu, 08 Jun 2017 15:06:25 GMTServer: nginxStrict-Transport-Security: max-age=31536000; includeSubDomains;Content-Length: 2008Connection: keep-aliveRegards,Seanattachment 1

starbright
Posts: 16
Joined: Sun Jul 31, 2005 9:10 pm America/New_York
Answers: 0

Direct Data Access, how to retrieve metadata?

by starbright » Thu Jun 08, 2017 2:30 pm America/New_York

Awesome! That's exactly what I was looking for! Thanks, Sean and Norman.

starbright
Posts: 16
Joined: Sun Jul 31, 2005 9:10 pm America/New_York
Answers: 0

Direct Data Access, how to retrieve metadata?

by starbright » Thu Jun 08, 2017 3:01 pm America/New_York

Sean,

One problem is that there seems no near real time data. For example, if I put this link in, it returns essentially empty results.

https://cmr.earthdata.nasa.gov/search/granules.json?provider=OB_DAAC&short_name=MODISA_L2_OC&page_size=100&page_num=1&polygon=10,10,30,10,30,20,10,20,10,10&temporal=2017-06-08T00:00:00Z,%202017-06-08T23:59:59Z

How could that be?

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1485
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Direct Data Access, how to retrieve metadata?

by OB.DAAC - SeanBailey » Thu Jun 08, 2017 3:17 pm America/New_York

That's because we don't put the data into CMR until we've processed the file as "refined" - which can be up to a two-week delay.
If you want NRT, create a subscription.

Sean

starbright
Posts: 16
Joined: Sun Jul 31, 2005 9:10 pm America/New_York
Answers: 0

Direct Data Access, how to retrieve metadata?

by starbright » Thu Jun 08, 2017 4:51 pm America/New_York

In the NRT data subscription, is there a way to specify a polygon as opposed to a bounding box?

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1485
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Direct Data Access, how to retrieve metadata?

by OB.DAAC - SeanBailey » Fri Jun 09, 2017 8:20 am America/New_York

No.  The information available for the search is rectilinear, so we stick to boxes on a sphere...

starbright
Posts: 16
Joined: Sun Jul 31, 2005 9:10 pm America/New_York
Answers: 0

Direct Data Access, how to retrieve metadata?

by starbright » Fri Jun 09, 2017 9:28 am America/New_York

I'm trying to think of a way to work around this limitation. If I create multiple boxes that partly overlap one another, how will your server handle the multiple boxes? Will your FTP server put a bunch of data granules pooled in one folder, or multiple folders (each corresponding to one box)? If the latter will be the case, I'm afraid there might be redundancies of data granules.

I could not do test subscriptions to answer the questions myself because I created a test subscription yesterday (with FTP as the option to receive data), but got no emails so far notifying me where to get the data.

I know it's a lot of questions. Thank you so much for the patience!

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1485
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Direct Data Access, how to retrieve metadata?

by OB.DAAC - SeanBailey » Fri Jun 09, 2017 9:55 am America/New_York

Yes, there will be redundant granules as each subscription is treated separately.
Just make a single request with a box big enough to cover your ROI polygon.

Sean

Post Reply