Missing files in GPM IMERG data

[earthengine_urs]

Hi,

CMR entries for GPM IMERG datasets occasionally miss files.

Example 1.

The files for 21:00 and 21:30 are present in https://gpm1.gesdisc.eosdis.nasa.gov/data/GPM_L3/GPM_3IMERGHHL.06/2020/238/, yet CMR does not know about them.

Compare 2 results here
https://cmr.earthdata.nasa.gov/search/granules.json?concept_id[]=C1598621095-GES_DISC&temporal[]=2020-08-25T20:30:00/2020-08-25T22:00:00&page_size=1000
with the previous day's 4 results
https://cmr.earthdata.nasa.gov/search/granules.json?concept_id[]=C1598621095-GES_DISC&temporal[]=2020-08-24T20:30:00/2020-08-24T22:00:00&page_size=1000

Example 2.

This skips files for 01:00 and 01:30
https://cmr.earthdata.nasa.gov/search/granules.json?online_only=true&collection_concept_id[]=C1598621095-GES_DISC&page_size=3&temporal=2022-12-29T00:00:00

Note how the file for 00:30
https://data.gesdisc.earthdata.nasa.gov/data/GPM_L3/GPM_3IMERGHHL.06/2022/363/3B-HHR-L.MS.MRG.3IMERG.20221229-S003000-E005959.0030.V06C.HDF5
is followed by the file for 02:00
https://data.gesdisc.earthdata.nasa.gov/data/GPM_L3/GPM_3IMERGHHL.06/2022/363/3B-HHR-L.MS.MRG.3IMERG.20221229-S020000-E022959.0120.V06C.HDF5

But https://jsimpsonhttps.pps.eosdis.nasa.gov/text/imerg/late/202212/ has these files.

[GES DISC - jimacker]

Thank you for bringing this to our attention. We will investigate the mismatching information in CMR.

[GES DISC - jimacker]

We have examined what you reported, and we do not see differences between the cumulus-published CMR inventory, and the inventory in s4pa for IMERGHHL. The latter is also reconciled with PPS (the jsimpson host).

There is nothing missing in CMR for IMERGHHL.

Users should be careful in using CMR API queries, and give sufficiently large page_size, and even give page_num, especially when globbing ([]) is used in the temporal constraints.
We recommend always using the EDSC interface to check the available inventory, and thus you can diagnose CMR API queries.

It’s worth noticing that there is latency between files appearing on the jsimpson host, then in s4pa (GES DISC), and then in CMR. That may give an apparent impression of two files from the leading edge missing in CMR, but that should resolve within an hour.

[earthengine_urs]

Thank you! (Sorry for the late reply.)

Yes, when I checked later, the discrepancies were gone.