Earthdata Forum

Hello! I am trying to interpret infromation from the MOD06_L2 data product and I had some questions I can't find answers to. Apologies if these are asked frequently, I could not find the solutions in my search.

First, the data is defined based on a scale value and an add offset per variable. Are these values fixed over time? I am not sure if these are determined based on the data range present in the specific granule or if they are designed to capture the full valid range of the variable.

The second data is about channels like Cloud_Optical_Thickness and Cloud_Optical_Thickness_PC, Cirrus_Reflectance, and a handful of other variables that seem to always contain NODATA for the granules I have looked at. I don't see any alerts that would indicate that these aren't being collected, so I am not sure what to expect. Should these be valid in certain date ranges? How can I determine when this data may be absent at search time?

Thank you for anyone willing to help!

Yes, the scale and the add_offset are fixed for each Scientific Data Set (SDS). The following conversion equation should be used to convert stored integers back to useful floating point values:

float_value = scale_factor * (integer_value – add_offset)

Information about missing data is provided in the product User Guide at:
https://atmosphere-imager.gsfc.nasa.gov/sites/default/files/ModAtmo/MODISCloudOpticalPropertyUserGuideFinal_v1.1_1.pdf

All known data issues are available at:
https://atmosphere-imager.gsfc.nasa.gov/products/cloud/data-issues

Hi! Thanks for the response.

Even after scouring the user's manual I am still struggling to identify when I should expect valid data for things like Cloud_Optical_Thickness. I would really like to distinguish between the no data case where the optical thinkness could confidently be considered zero, vs the no data case where there was some kind of sensor or algorithmic failure. I could potentially reconstruct a mask based on the failure or QA flags, but each field in these seems very specific and I am not confident that my mask would be accurate. Even if I did go down that route it doesn't look like software packages like GDAL really support that kind of mutli-bitmask approach. I would end up with a pretty sophisticated processing layer to try and answer a simple question, which makes me think that I am missing something as to how this data is used in practice.

If anyone has any experience with this I would appreciate any guidance.

Our product SME investigated MOD06_L2 data and found how to distinguish Cloud_Optical_Thickness (COT) between no data case and failed retrieval.

For example, for 2.1 micron (band 7), the valid range for scaled Cloud_Optical_Thickness (1354, 2030) is [0,150]. The valid range for scaled Retrieval_Failure_Metric is [0, 200].

1. Failed retrieval case:

The following two cells have fill value for COT, but Retrieval_Failure_Metric has valid value, which means missing data in COT is caused by some kind of sensor or algorithmic failure.
Cloud_Optical_Thickness(17,1) = NaN
Cloud_Optical_Thickness(18, 1) = NaN

Retrieval_Failure_Metric(17, 1) = 0.1
Retrieval_Failure_Metric(18, 1) = 0.1

2. No COT data case:

Cloud_Optical_Thickness(1,1) = NaN
Cloud_Optical_Thickness(10, 1) = NaN

Retrieval_Failure_Metric(1, 1) = NaN
Retrieval_Failure_Metric(10, 1) = NaN

Thanks for the help! I am seeing the Retrieval_Failure_Metric SDS with dimensions (3,1354,3) and I am not sure how to match that to the Cloud_Optical_Thickness demensions (1354, 2030). Based on the manual it seems that there are COT, CER, and cost metrics, which explains one of the 3 dimensions, but I don't understand why the second dimension is 3 rather than 2030. Much of the manual is a bit over my head, so I appreciate the patience.

Here is the description for the 3-dimension “Retrieval_Failure_Metric” in a MOD06_L2 granule:

short Retrieval_Failure_Metric(Cell_Along_Swath_1km=2030, Cell_Across_Swath_1km=1354, RFM_nband=3);
:long_name = "Retrievals and other information for points that failed to retrieve via standard solution logic for retrieval using band 7 and either band 1, 2, or 5 (specified in Quality_Assurance_1km)";
:Cell_Across_Swath_Sampling = 1, 1354, 1; // int
:Cell_Along_Swath_Sampling = 1, 2030, 1; // int
:units = "by plane in order: 1-none, 2-micron, 3-percent";

The third dimension is for 3 elements retrieval: 1—Cloud Optical Thickness; 2 – Cloud Effective Radius; 3—Relative distance of failed retrieval from the library space as percentage of reflectance.


Retrieval_Failure_Metric is for 2.1 micron (Band 7)
Retrieval_Failure_Metric_16 is for 1.6 micron (Band 6)
Retrieval_Failure_Metric_37 is for 3.7 micron (Band 20)
Retrieval_Failure_Metric_1621 is for 1.6 – 2.1 micron (Band 6 – Band 7)

Please see section 2.6 (Table 2.6-1) in User Guide for the Collection 6/6.1 L2 MOD06/MYD06 Product (Version 1.1, July 2018) for more details.

https://atmosphere-imager.gsfc.nasa.gov/sites/default/files/ModAtmo/MODISCloudOpticalPropertyUserGuideFinal_v1.1_1.pdf

It sounds like you are saying the dimensions I am seeing don't match the expectations for the band? I am currently using gdal to process the data, is there a possibility I am running into a bug with that software package? I am currently looking at MYD06_L2.A2022335.0445.061.2022336010921.hdf and have seen the same structure for a number of files around that time period from MYD and MOD. This is also the structure for the band specific SDS as well.

Getting the shape in python:

In [48]: cot = gdal.Open('HDF4_EOS:EOS_SWATH:"MYD06_L2.A2022335.0445.061.2022336010921.hdf":mod06:Retrieval_Failure_Metric')
In [49]: arr = cot.ReadAsArray()
In [50]: arr.shape
Out[50]: (3, 1354, 3)

Here is my gdalinfo output for the SDS.

gdalinfo HDF4_EOS:EOS_SWATH:"MYD06_L2.A2022335.0445.061.2022336010921.hdf":mod06:Retrieval_Failure_Metric

Driver: HDF4Image/HDF4 Dataset
Files: MYD06_L2.A2022335.0445.061.2022336010921.hdf
Size is 3, 1354
Metadata:
add_offset=0
ALGORITHMPACKAGEACCEPTANCEDATE=June 1997
ALGORITHMPACKAGEMATURITYCODE=at-launch
ALGORITHMPACKAGENAME=ATBD-MOD-04 and ATBD-MOD-05
ALGORITHMPACKAGEVERSION=2
ALGORITHM_VERSION_CLOUD_PHASE_IR=1
ALGORITHM_VERSION_CLOUD_PROPERTY_VIS=1
ALGORITHM_VERSION_CLOUD_TOP_PROPERTY_IR=1
ANCILLARYINPUTPOINTER.1=MYD03.A2022335.0445.061.2022335161759.hdf
ANCILLARYINPUTTYPE.1=Geolocation
ASSOCIATEDINSTRUMENTSHORTNAME.1=MODIS
ASSOCIATEDPLATFORMSHORTNAME.1=Aqua
ASSOCIATEDSENSORSHORTNAME.1=MODIS
AUTOMATICQUALITYFLAG.1=Passed
AUTOMATICQUALITYFLAGEXPLANATION.1=Passed: >10% useable; Failed: <10% useable
Cell_Across_Swath_Sampling=1, 1354, 1
Cell_Along_Swath_Sampling=1, 2030, 1
CirrusCloudDetectedPct_IR= 59.77
Clear_Sky_Restoral_Status=y
CloudPhaseUncertainPct_IR= 33.53
Collection_4_Phase_Used=n
DAYNIGHTFLAG=Night
description=\n

The individual planes in this SDS are set to contain the following:
1 -- Cloud Optical Thickness
2 -- Cloud Effective Radius
3 -- Relative distance of failed retrieval from the library space as
percentage of reflectance

DESCRREVISION=6.1
EASTBOUNDINGCOORDINATE=22.7361626004592
EQUATORCROSSINGDATE.1=2022-12-01
EQUATORCROSSINGLONGITUDE.1=-51.292413351383
EQUATORCROSSINGTIME.1=05:07:38.301730
EXCLUSIONGRINGFLAG.1=N
Geolocation_Pointer=External MODIS geolocation product
GRINGPOINTLATITUDE.1=82.2476036095849, 69.8630108991823, 58.1260939642675, 64.2336772519593
GRINGPOINTLONGITUDE.1=-66.4540035112062, 22.7424611344673, -10.4213125885142, -54.0782627568564
GRINGPOINTSEQUENCENO.1=1, 2, 3, 4
HDFEOSVersion=HDFEOS_V2.19
HighCloudDetectedPct_IR= 50.00
history=$Id: MOD06_L2.CDL.fs,v 1.13 2013/06/19 15:38:46 wind Exp $

IceCloudDetectedPct_IR= 42.95
Ice_Phase_Forced=n
identifier_product_doi=10.5067/MODIS/MYD06_L2.061
identifier_product_doi=10.5067/MODIS/MYD06_L2.061
identifier_product_doi_authority=https://doi.org
identifier_product_doi_authority=https://doi.org
INPUTPOINTER=MYD021KM.A2022335.0445.061.2022335180830.hdf, MYD03.A2022335.0445.061.2022335161759.hdf, MYD35_L2.A2022335.0445.061.2022335180915.hdf, gdas1.PGrbF00.221201.06z, oisst.20221123, goge1_2_img.v1, modisdet.dry.101.lit_end.v3, modisdet.ozo.101.lit_end.v3, modisdet.wts.101.lit_end.v3, modisdet.wtl.101.lit_end.v3, modisdet.wco.101.lit_end.v3, MYDCSR_B.A2022327.061.2022336000952.hdf, gdas1.PGrbF00.221201.00z, eng.221201, NISE_SSMISF18_20221201.HDFEOS, MODIS_Ice_library.hdf.v5, MODIS_Water_library.hdf.v4, MODIS_Ice_library_ws3.hdf.v3, MODIS_Ice_library_ws3sd.hdf.v4, MODIS_Ice_library_ws7.hdf.v3, MODIS_Ice_library_ws7sd.hdf.v4, MODIS_Ice_library_ws15.hdf.v3, MODIS_Ice_library_ws15sd.hdf.v4, MODIS_Water_library_ws3.hdf.v2, MODIS_Water_library_ws3sd.hdf.v3, MODIS_Water_library_ws7.hdf.v2, MODIS_Water_library_ws7sd.hdf.v3, MODIS_Water_library_ws15.hdf.v2, MODIS_Water_library_ws15sd.hdf.v3, MODIS_Ice_WaterPhaseFunc.hdf.v3, Transmittance.hdf.v2, IGBP.EcoMap.NtoS.2004.149.v004.hdf, AlbSnwStst.ByNISE.W90.D90.WS.Hemi.2000-2004.YrAvg.hdf, MCD43GF_wsa_Band1_329_2016_V006.hdf, MCD43GF_wsa_Band2_329_2016_V006.hdf, MCD43GF_wsa_Band5_329_2016_V006.hdf, MCD43GF_wsa_Band6_329_2016_V006.hdf, MCD43GF_wsa_Band7_329_2016_V006.hdf
INSTRUMENTNAME=Moderate Resolution Imaging Spectroradiometer
LandCoverFractionPct= 36.37
LOCALGRANULEID=MYD06_L2.A2022335.0445.061.2022336010921.hdf
LOCALINPUTGRANULEID=MYD021KM.A2022335.0445.061.2022335180830.hdf, MYD03.A2022335.0445.061.2022335161759.hdf, MYD35_L2.A2022335.0445.061.2022335180915.hdf, MYDCSR_B.A2022327.061.2022336000952.hdf
LOCALVERSIONID=061
LONGNAME=MODIS/Aqua Clouds 5-Min L2 Swath 1km and 5km
long_name=Retrievals and other information for points that failed to retrievevia standard solution logic for retrieval using band 7 and either band 1, 2, or 5 (specified in Quality_Assurance_1km)
LowCloudDetectedPct_IR= 27.67
Maximum_Number_of_1km_Frames=1354
MidCloudDetectedPct_IR= 11.89
MixedCloudDetectedPct_IR= 0.00
NORTHBOUNDINGCOORDINATE=82.6753205054503
Number_of_Instrument_Scans=2030
OceanCoverFractionPct= 63.63
OpaqueCloudDetectedPct_IR= 29.79
ORBITNUMBER.1=109452
PARAMETERNAME.1=Cloud_Top_Pressure
Parameter_Type=Output
PGEVERSION=6.1.9
PROCESSINGENVIRONMENT=Linux minion20237 5.4.0-1064-fips #73-Ubuntu SMP Mon Oct 17 18:45:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
PRODUCTIONDATETIME=2022-12-02T01:09:21.000Z
PRODUCTIONHISTORY=PGE06:6.1.9
QAPERCENTMISSINGDATA.1=10
RANGEBEGINNINGDATE=2022-12-01
RANGEBEGINNINGTIME=04:45:00.000000
RANGEENDINGDATE=2022-12-01
RANGEENDINGTIME=04:50:00.000000
REPROCESSINGACTUAL=processed once
REPROCESSINGPLANNED=further update is anticipated
scale_factor=0.00999999977648258
SCIENCEQUALITYFLAG.1=Not Investigated
SCIENCEQUALITYFLAGEXPLANATION.1=See http://modis-atmos.gsfc.nasa.gov/validation.html for more details on MODIS Atmosphere data quality.
SHORTNAME=MYD06_L2
SnowCoverFractionPct= 47.70
SOUTHBOUNDINGCOORDINATE=58.3595385309713
SuccessCloudPhaseRtrPct_IR= 89.20
SuccessCloudTopPropRtrPct_IR= 89.56
ThickCloudDetectedPct_IR= 40.08
ThinCloudDetectedPct_IR= 19.69
title=MODIS Level 2 Cloud Properties

units=by plane in order: 1-none, 2-micron, 3-percent
valid_range=0, 20000
VERSIONID=61
WaterCloudDetectedPct_IR= 12.72
Water_Phase_Forced=n
WESTBOUNDINGCOORDINATE=-64.0647212102571
_FillValue=-9999
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 1354.0)
Upper Right ( 3.0, 0.0)
Lower Right ( 3.0, 1354.0)
Center ( 1.5, 677.0)
Band 1 Block=3x1354 Type=Int16, ColorInterp=Gray
Description = Retrievals and other information for points that failed to retrieve via standard solution logic for retrieval using band 7 and either band 1, 2, or 5 (specified in Quality_Assurance_1km)
NoData Value=-9999
Unit Type: by plane in order: 1-none, 2-micron, 3-percent
Offset: 0, Scale:0.00999999977648258
Band 2 Block=3x1354 Type=Int16, ColorInterp=Gray
Description = Retrievals and other information for points that failed to retrieve via standard solution logic for retrieval using band 7 and either band 1, 2, or 5 (specified in Quality_Assurance_1km)
NoData Value=-9999
Unit Type: by plane in order: 1-none, 2-micron, 3-percent
Offset: 0, Scale:0.00999999977648258
Band 3 Block=3x1354 Type=Int16, ColorInterp=Gray
Description = Retrievals and other information for points that failed to retrieve via standard solution logic for retrieval using band 7 and either band 1, 2, or 5 (specified in Quality_Assurance_1km)
NoData Value=-9999
Unit Type: by plane in order: 1-none, 2-micron, 3-percent
Offset: 0, Scale:0.00999999977648258

We cannot provide assistance with gdal. However, you can see the complete file specification of the product which gives a complete description of the Cloud (06_L2) HDF product file, including all the Scientific Data Sets and their local attributes at:
https://atmosphere-imager.gsfc.nasa.gov/products/cloud/file-spec
You can also find examples of code in various languages, to access and visualize product files:
https://www.hdfeos.org/zoo/index_openLAADS_Examples.php#MODIS

Ok, yeah, HDFView shows these retrieval failures with the correct resolution on the same files. Seems like I was beating my head agaisnt a broken software package. I will give pyhdf a try based on your suggestion.

Thanks for all the help! I'm back on track.