UUI subsetting "datasetID" - how to find?

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
tkurosu
Posts: 3
Joined: Mon Apr 12, 2021 6:18 pm America/New_York
Answers: 0

UUI subsetting "datasetID" - how to find?

by tkurosu » Mon Apr 12, 2021 6:31 pm America/New_York

is there either a method or a central location with/from which to obtain the exact "datasetID" string that is required for a subsetting query? for example, i am trying to get data subsets of the SNPP OMPS-NM Total Ozone product, but the subsetID appears to be neither "OMPS_NPP_NMTO3_L2" (product page title), nor "OMPS_NPP_NMTO3-L2_v2.1" nor "OMPS-NPP_NMTO3-L2_v2.1" (product file name) nor any combination of "-" and "_" of those. the generic response "jsonwsp/fault" from the Python-based query run is not helpful, obviously. i have a range of different products from various sensors i want to subset-download, and guessing at what the datasetID might be is frustrating. ThanX!

Tags:

GES DISC - jimacker
User Services
User Services
Posts: 299
Joined: Mon Sep 30, 2019 1:57 pm America/New_York
Answers: 3
Has thanked: 2 times
Been thanked: 4 times

Re: UUI subsetting "datasetID" - how to find?

by GES DISC - jimacker » Tue Apr 13, 2021 11:06 am America/New_York

Thanks for your question. I will forward it to our developers for comment.

tkurosu
Posts: 3
Joined: Mon Apr 12, 2021 6:18 pm America/New_York
Answers: 0

Re: UUI subsetting "datasetID" - how to find?

by tkurosu » Tue Apr 13, 2021 12:13 pm America/New_York

thanks! the particular data set i mentioned (OMPS_NPP_NMTO3-L2_v2.1) may not be "subset-able", since unlike for some other products there doesn't appear to be a "Subset" option next to the "Get Data" tab. however, the general question remains. it may be possible to infer the datasetID from the URL of a product, but it would be very helpful for non-interactive queries to be able to check a list of subset-able products and their datasetID.

GES DISC - jimacker
User Services
User Services
Posts: 299
Joined: Mon Sep 30, 2019 1:57 pm America/New_York
Answers: 3
Has thanked: 2 times
Been thanked: 4 times

Re: UUI subsetting "datasetID" - how to find?

by GES DISC - jimacker » Wed Apr 14, 2021 3:38 pm America/New_York

Hello,

Here is the answer provided by our GES DISC staff.

There are two ways to get the DatasetId.

1. The search API (https://disc.gsfc.nasa.gov/information/ ... 0Searching) does provide a way to get the DatasetID; here is the code snippet from that HowTo article that prints out the DatasetID and the Label, a more human-readable description of the dataset:
# Report on the results: DatasetID and Label
if total > 0 :
for item in response['result']['items']:
print('%-20s %s' % (item['dataset']['id'], item['dataset']['label']))
2. An easy but non-programmatic option is to use our web site and go through the “Subset / Get Data” interface. Use the subset option accordions to set up the desired subset constraints, and then click on the “Get Data” button. When list of URLs to download appears, click on the small JobID link in the bottom right corner of the dialog box and it will show you what the syntax of your API request should look like, including the datasetID and any other relevant parameters.

For example, a spatial and variable subset of GPM IMERG half-hourly data using OPeNDAP:
This is the JobID link: https://disc.gsfc.nasa.gov/api/jobs/arg ... 0dfc100df2

This is what it shows (carriage returns inserted for clarity):
{"box":[-130,22,-60,52],
"crop":true,
"agent":"OPeNDAP",
"role":"subset",
"data":[{"datasetId":"GPM_3IMERGHH_06",
"variable":"precipitationCal"}],
"start":"2000-06-01T00:00:00Z",
"end":"2021-04-14T12:20:26Z",
"format":"netCDF4"}

tkurosu
Posts: 3
Joined: Mon Apr 12, 2021 6:18 pm America/New_York
Answers: 0

Re: UUI subsetting "datasetID" - how to find?

by tkurosu » Mon May 10, 2021 3:00 pm America/New_York

i can get the API script to work (copy/paste to a file called gesdisc_ss_test.py), but for the data sets i am interested in, the S5P_L1B_RA_BD1_HiR_1 (and equivalent for BD2 and BD3), it throws an error. i call the script from the command line with "python gesdic_ss_test.py", then enter search terms as suggested on the API HowTo page. a search for

s5p l1 bd1

succeeds (as do equivalent searches for bd2 and bd3), but

s5p l1 bd1 hir

throws the following error in the "# Report on the results: DatasetID and variable subsetting information/# Check for subset services" block:
Traceback (most recent call last):
File "gesdisc_ss_test.py", line 77, in <module>
if item['services']['subset']:
KeyError: 'subset'

this appears to be related to a non-subsettable HiR product, since screen I/O contains the lines

The OPeNDAP service supports variable subsetting for S5P_L1B_RA_BD1_HiR_1
The OPeNDAP service supports variable subsetting for S5P_L1B_RA_BD1_1

however, no actual data query for either S5P_L1B_RA_BD1_1 or S5P_L1B_RA_BD1_HiR_1 (and similar for BD2 and BD3) appear to be possible. in my set-up, a query for datasetID OCO2_L2_Lite_FP_9r succeeds, but changing that to any of the S5P L1B id strings returns "API Error: faulty request". the query places no constraints on variable names ("*"), the requested time period is covered by all products, and no other OCO2-specific constraints are given. since the query succeeds for OCO2, i can rule out errors in the set-up. i am stumped.

GES DISC - jimacker
User Services
User Services
Posts: 299
Joined: Mon Sep 30, 2019 1:57 pm America/New_York
Answers: 3
Has thanked: 2 times
Been thanked: 4 times

Re: UUI subsetting "datasetID" - how to find?

by GES DISC - jimacker » Thu May 13, 2021 12:44 pm America/New_York

We will examine your question and try to determine a solution. Please stand by.

Post Reply