Earthdata CMR WIP?

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
b.detracey
Posts: 38
Joined: Tue Sep 29, 2015 11:10 am America/New_York
Answers: 0

Earthdata CMR WIP?

by b.detracey » Tue Nov 07, 2017 11:01 am America/New_York

Hi,
I recently noticed the SeaDAS python script fd_matchup.py which led me to the Earthdata CMR search API. The following collection search for OB.DAAC L1/L2 datasets gives:
$ wget -q "https://cmr.earthdata.nasa.gov/search/collections.xml?page_size=2000&provider=OB_DAAC\
&processing_level_id=1&processing_level_id=2&pretty=true" -O - | grep name | sort
            <name>CZCS_L1</name>
            <name>CZCS_L2_OC</name>
            <name>GOCI_L1</name>
            <name>GOCI_L2_OC</name>
            <name>HICO_L1</name>
            <name>MERIS_L1_RR</name>
            <name>MERIS_L2_IOP</name>
            <name>MERIS_L2_OC</name>
            <name>MODISA_L2_IOP</name>
            <name>MODISA_L2_LAND</name>
            <name>MODISA_L2_OC</name>
            <name>MODISA_L2_SST</name>
            <name>MODIST_L2_IOP</name>
            <name>MODIST_L2_OC</name>
            <name>MODIST_L2_SST</name>
            <name>OCTS_L1</name>
            <name>OCTS_L2_IOP</name>
            <name>OCTS_L2_OC</name>
            <name>S3A_OLCI_L1</name>
            <name>SeaWiFS_L1</name>
            <name>SeaWiFS_L2_IOP</name>
            <name>SeaWiFS_L2_LAND</name>
            <name>SeaWiFS_L2_OC</name>
            <name>VIIRSN_L2_IOP</name>
            <name>VIIRSN_L2_LAND</name>
            <name>VIIRSN_L2_OC</name>
            <name>VIIRSN_L2_SST</name>

I have two questions:
Are there plans to add the MODIS and VIIRS L1 metadata to the CMR?
At present there seems to be no way, on the collection level, to discriminate between full and reduced resolution MERIS, between full and reduced resolution OLCI, or between GAC and MLAC SeaWiFS. Are there plans to allow discrimination of these as separate collections? Or should I 'manually' discriminate by filtering the results of a granule search?

Sorry. I guess that was three questions... :eek:

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Earthdata CMR WIP?

by OB.DAAC - SeanBailey » Mon Nov 13, 2017 9:08 am America/New_York

1) Maybe.  Currently, the MODIS L1 data are in CMR, but cataloged by MODAPS/LAADS.   I had thought they would do the same for VIIRS...
2) No
3) Yes

Regards,
Sean

b.detracey
Posts: 38
Joined: Tue Sep 29, 2015 11:10 am America/New_York
Answers: 0

Earthdata CMR WIP?

by b.detracey » Tue Nov 21, 2017 8:38 am America/New_York

Got it. Thanks Sean.
Searching on Earthdata the only L1 VIIRS product is VIIRS/NPP Moderate Resolution Terrain Corrected Geolocation 6-Min L1 Swath 750m Light V001. It would be helpful if your MODIS L1 assets were added as my understanding is that they are not the same as those of MODAPS/LAADS.

A few (hopefully) last questions, mostly directed at understanding whether a CMR query can tell me if a collection and/or granule has been fixed/reprocessed/updated.
-It looks like only your refined products are in the CMR. Makes sense to me but could you please verify?
-Within CMR, will the granule search parameter 'updated_since' or 'revision_date' tell me if a granule itself has been updated? Or can this not discriminate from a metadata update? I am trying to understand if CMR can be used to determine, at my year's end reprocessing, whether individual granules have been altered/fixed by your group. Or does it make more sense to stick with checksums from your own file search api?
-What is the reason that the CMR granule metadata "Granule/Collection/VersionID" matches only the major processing version and not the netcdf pversion attribute?

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Earthdata CMR WIP?

by OB.DAAC - SeanBailey » Tue Nov 21, 2017 4:05 pm America/New_York

The MODIS L1A files we distribute are identical to the LAADS data.  L1B is not, but we don't distribute L1B (LAADS does, though).

Yes, currently we only put the refined data in CMR.
We generally do not alter files between reprocessings (except for the quicklook/refined delineation, but as we're only putting refined in CMR, a non-issue).
The collection versionID refers to a consistent set.  The sub versions in the netCDF pversion attribute typically are just calibration updates that affect a subset of the data.
The calibration is consistent within the larger collection, but we may have had to reprocess a portion of the data to ensure that consistency.

Sean

b.detracey
Posts: 38
Joined: Tue Sep 29, 2015 11:10 am America/New_York
Answers: 0

Earthdata CMR WIP?

by b.detracey » Mon Jun 04, 2018 1:55 pm America/New_York

I have noticed the OB DAAC metadata on CMR is in flux:
$ wget -q "https://cmr.earthdata.nasa.gov/search/collections.xml?page_size=2000&provider=OB_DAAC\
&processing_level_id=2&pretty=true" -O - | grep name | sort
            <name>ADEOS-I Ocean Color and Temperature Scanner (OCTS) Inherent Optical Properties (IOP) Regional  Data</name>
            <name>ADEOS-I Ocean Color and Temperature Scanner (OCTS) Ocean Color (OC) Regional  Data</name>
            <name>Aqua Moderate Resolution Imaging Spectroradiometer (MODIS)  Inherent Optical Properties (IOP) - NRT</name>
            <name>Aqua Moderate Resolution Imaging Spectroradiometer (MODIS)  Ocean Color (OC) - NRT</name>
            <name>Aqua Moderate Resolution Imaging Spectroradiometer (MODIS) Inherent Optical Properties (IOP) Regional  Data</name>
            <name>Aqua Moderate Resolution Imaging Spectroradiometer (MODIS) Ocean Color (OC) Regional  Data</name>
            <name>COMS Geostationary Ocean Color Imager (GOCI) Ocean Color (OC) Regional  Data</name>
            <name>MODISA_L2_SST</name>
            <name>Nimbus-7 Coastal Zone Color Scanner (CZCS) Ocean Color (OC) Regional  Data</name>
            <name>NPP Visible Infrared Imager-Radiometer Suite (VIIRS) Inherent Optical Properties (IOP) Regional Data</name>
            <name>NPP Visible Infrared Imager-Radiometer Suite (VIIRS) Ocean Color (OC) Regional Data</name>
            <name>OrbView-2 Sea-Viewing WIde Field-of-view Sensor (SeaWiFS) Inherent Optical Properties (IOP) Regional  Data</name>
            <name>OrbView-2 Sea-Viewing WIde Field-of-view Sensor (SeaWiFS) Ocean Color (OC) Regional  Data</name>
            <name>Terra Moderate Resolution Imaging Spectroradiometer (MODIS)  Inherent Optical Properties (IOP) - NRT</name>
            <name>Terra Moderate Resolution Imaging Spectroradiometer (MODIS)  Ocean Color (OC) - NRT</name>
            <name>Terra Moderate Resolution Imaging Spectroradiometer (MODIS) Inherent Optical Properties (IOP) Regional Data, reprocesing v2018</name>
            <name>Terra Moderate Resolution Imaging Spectroradiometer (MODIS) Inherent Optical Properties (IOP) Regional Data, reprocesing v2018</name>
            <name>Terra Moderate Resolution Imaging Spectroradiometer (MODIS) Ocean Color (OC) Regional Data, reprocesing v2018</name>
            <name>Terra Moderate Resolution Imaging Spectroradiometer (MODIS) Ocean Color (OC) Regional Data, reprocesing v2018</name>
            <name>Terra Moderate Resolution Imaging Spectroradiometer (MODIS) Sea Surface Temperature (SST) Regional  Data</name>
            <name>Visible Infrared Imager-Radiometer Suite NPP Level-2 Normalized Difference Vegetation Index Land Reflectance Data Version 2014</name>


Any guess-timate on when it will be more flux-less (less fluxly)? I see also the addition of "NRT" vs "Regional Data". I thought only refined data was going in. :surprised:

OB.DAAC - amscott
User Services
User Services
Posts: 345
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Earthdata CMR WIP?

by OB.DAAC - amscott » Mon Jun 04, 2018 2:27 pm America/New_York

The decision was made to put the near-realtime data into CMR which are later replaced with the refined granules. GOCI is listed as regional since it is a geostationary satellite. L2 products are classified as regional as well but the L3 are global.  There will be flux for the foreseeable future as the mapping is changed from a geobox to a g-ring system and we continue the V2018 reprocessing of the remaining missions.

Tommy

b.detracey
Posts: 38
Joined: Tue Sep 29, 2015 11:10 am America/New_York
Answers: 0

Earthdata CMR WIP?

by b.detracey » Thu Jun 07, 2018 2:29 pm America/New_York

Head's up. MODISA_L2_SST_2014NRT is listed as level 3.
      "native-id" : "MODISA_L2_SST_2014NRT",
      "provider-id" : "OB_DAAC",
      "concept-type" : "collection",
      "concept-id" : "C1519120045-OB_DAAC",
      "has-spatial-subsetting" : false,
      "revision-date" : "2018-05-18T14:29:24Z",
      "user-id" : "obdaac_admin",
      "deleted" : false,
      "revision-id" : 6,
      "has-transforms" : false,
      "has-formats" : false,
      "format" : "application/echo10+xml"
    },
    "umm" : {
      "SpatialExtent" : {
        "SpatialCoverageType" : "HORIZONTAL",
        "HorizontalSpatialDomain" : {
          "Geometry" : {
            "CoordinateSystem" : "CARTESIAN",
            "BoundingRectangles" : [ {
              "WestBoundingCoordinate" : -180.0,
              "NorthBoundingCoordinate" : 90.0,
              "EastBoundingCoordinate" : 180.0,
              "SouthBoundingCoordinate" : -90.0
            } ]
          }
        },
        "GranuleSpatialRepresentation" : "CARTESIAN"
      },
      "Distributions" : [ {
        "DistributionFormat" : "netCDF4"
      } ],
      "CollectionProgress" : "IN WORK",
      "ScienceKeywords" : [ {
        "Category" : "Earth Science",
        "Topic" : "Oceans",
        "Term" : "Ocean Optics",
        "VariableLevel1" : "Absorption"
      } ],
      "TemporalExtents" : [ {
        "EndsAtPresentFlag" : false,
        "RangeDateTimes" : [ {
          "BeginningDateTime" : "2002-07-04T00:00:00.000Z"
        } ]
      } ],
      "ProcessingLevel" : {
        "Id" : "3"
      },
      "ShortName" : "MODISA_L2_SST_NRT",

OB.DAAC - amscott
User Services
User Services
Posts: 345
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Earthdata CMR WIP?

by OB.DAAC - amscott » Thu Jun 07, 2018 2:33 pm America/New_York

Corrected! Thanks.

Post Reply