Problem reading HDF5 SAGE-III/ISS files

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
vaqueromartinez
Posts: 1
Joined: Mon Nov 14, 2022 6:04 am America/New_York
Answers: 0

Problem reading HDF5 SAGE-III/ISS files

by vaqueromartinez » Mon Nov 14, 2022 8:20 am America/New_York

I downloaded some SAGE-III/ISS files with water vapor data. These are HDF files, but I think I am not reading them properly with command line "h5dump" or R package "rhdf5". In both cases, I get what looks some kind of integer overflow on some QA flags: I get the number 2147483647 (2^31 - 1). The QA variables are supposed to be 32-bit integers with some bytes showing 0-1 flags. I think I get some overflow values with float variables too.

As an example, I show here the H2O_QA variable I get:

> h5dump -d "/Altitude Based Data/Water Vapor Profiles/H2O_QA" g3b.ssp.2022061601SRv05.21
HDF5 "g3b.ssp.2022061601SRv05.21" {
DATASET "/Altitude Based Data/Water Vapor Profiles/H2O_QA" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 200 ) / ( 200 ) }
DATA {
(0): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(5): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(10): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(15): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(20): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(25): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(30): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(35): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(40): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(45): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(50): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(55): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(60): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(65): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(70): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(75): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(80): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(85): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(90): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(95): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(100): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(105): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(110): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(115): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(120): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(125): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(130): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(135): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(140): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(145): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(150): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(155): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(160): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(165): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(170): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(175): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(180): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(185): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(190): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647,
(195): 2147483647, 2147483647, 2147483647, 2147483647, 2147483647
}
ATTRIBUTE "DIMENSION_LABELS" {
DATATYPE H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): "Altitude"
}
}
ATTRIBUTE "DIMENSION_LIST" {
DATATYPE H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT }}
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): ()
}
}
ATTRIBUTE "_FillValue" {
DATATYPE H5T_STD_I32LE
DATASPACE SCALAR
DATA {
(0): 2147483647
}
}
ATTRIBUTE "bit_description" {
DATATYPE H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_UTF8;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "Bits 0-3 smoothing kernel used for shell (0 No smoothing, 1 1-2-1 smoothing, 2 1-2-3-2-1 smoothing, 3 5pt boxcar average, 4 7pt boxcar average, 5 9pt boxcar average, 6 11pt boxcar average). Bit 4 retrieved slant-path profile value was negative. Bit 5 retrieved slant-path profile value contained \37777777742\37777777600\37777777630fill\37777777742\37777777600\37777777631 data. Bit 6 shell was given the \37777777742\37777777600\37777777630fill\37777777742\37777777600\37777777631 value in the smoothing process because the shell was outside of the altitude window selected for smoothing."
}
}
ATTRIBUTE "long_name" {
DATATYPE H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_UTF8;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
(0): "H2O Concentration QA Flag"
}
}
}
}

Tags:

ASDC - cheyenne.e.land
Subject Matter Expert
Subject Matter Expert
Posts: 129
Joined: Mon Mar 22, 2021 3:55 pm America/New_York
Answers: 1
Has thanked: 1 time
Been thanked: 8 times

Re: Problem reading HDF5 SAGE-III/ISS files

by ASDC - cheyenne.e.land » Mon Nov 14, 2022 10:02 am America/New_York

Hello,

A Subject Matter expert has been notified and will answer your question as soon as possible.

ASDC User Services

ASDC - cheyenne.e.land
Subject Matter Expert
Subject Matter Expert
Posts: 129
Joined: Mon Mar 22, 2021 3:55 pm America/New_York
Answers: 1
Has thanked: 1 time
Been thanked: 8 times

Re: Problem reading HDF5 SAGE-III/ISS files

by ASDC - cheyenne.e.land » Mon Nov 14, 2022 11:29 am America/New_York

Hello,

After the SAGE III-ISS team briefly reviewed your comment they believe you are working correctly.
As mentioned in page 15 of the data products users guide the Retrieved Profile QA Flags are set to integer fill value for v5.2x.
This will be remedied in the next version.

Hope this successfully answers the questions.

Regards,
ASDC User Services

Post Reply