Missing checksums in VIIRS L3U products

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
richli
Posts: 6
Joined: Mon Sep 25, 2023 5:11 pm America/New_York
Answers: 0

Missing checksums in VIIRS L3U products

by richli » Mon Jan 29, 2024 4:48 pm America/New_York

Lately, I've noticed some missing checksums in the CMR responses for some VIIRS granules in the VIIRS-NPP L3U [1] and VIIRS-N20 L3U [2] datasets.

Most granules have a CMR response that look like [3]:

[
{
"SizeUnit": "MB",
"Size": 5.189972877502441,
"Checksum": {
"Value": "8d9d442d4bed696eb35fcb5446c541e1",
"Algorithm": "MD5"
},
"SizeInBytes": 5442081,
"Name": "20231107072000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc"
},
{
"SizeUnit": "MB",
"Size": 0.0001087188720703125,
"Checksum": {
"Value": "40a5dd36f72bfc0f3e85e85a3347deef",
"Algorithm": "MD5"
},
"SizeInBytes": 114,
"Name": "20231107072000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc.md5"
}
]

But some granules are missing the "Checksum" item, like [4]:

[
{
"SizeUnit": "MB",
"Size": 0.8209867477416992,
"SizeInBytes": 860867,
"Name": "20231207072000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc"
}
]

Could the missing MD5 sums be included?

Here's a list of the granule filenames over the past two months I've found that have missing checksums:

20231204221000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20231205194000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20231207072000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20231216043000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20231217083000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20231218173000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20231218193000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20231231012000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240125192000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240125145000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240126092000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240128135000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc

[1] https://podaac.jpl.nasa.gov/dataset/VIIRS_NPP-STAR-L3U-v2.80
[2] https://podaac.jpl.nasa.gov/dataset/VIIRS_N20-STAR-L3U-v2.80
[3] curl -sS 'https://cmr.earthdata.nasa.gov/search/concepts/G2797214324-POCLOUD' | jq '.DataGranule.ArchiveAndDistributionInformation'
[4] curl -sS 'https://cmr.earthdata.nasa.gov/search/concepts/G2813132483-POCLOUD' | jq '.DataGranule.ArchiveAndDistributionInformation'

Thanks! (And apologies for the poor formatting, I don't know how to enable BBCode in this post.)
Filters:
Data Access
Earthdata Search
Ocean
PODAAC
VIIRS
NOAA-20/JPSS-1
Top
PODAAC - wenhaoli
Subject Matter Expert
Subject Matter Expert
Posts: 35
Joined: Tue May 11, 2021 12:58 pm America/New_York
Answers: 0

Re: Missing checksums in VIIRS L3U products

by PODAAC - wenhaoli » Fri Feb 02, 2024 7:04 pm America/New_York

Thank you. We are looking into this issue and will get back to you soon.
Top
PODAAC - wenhaoli
Subject Matter Expert
Subject Matter Expert
Posts: 35
Joined: Tue May 11, 2021 12:58 pm America/New_York
Answers: 0

Re: Missing checksums in VIIRS L3U products

by PODAAC - wenhaoli » Wed Mar 20, 2024 3:06 pm America/New_York

Sorry for the delay. Finally, we have fixed the missing md5 file issue, which were occurred during the data ingest transition from on-prem to s3 bucket. All the missing md5 files are uploaded again. Please let us know if you have any questions.

Thanks
Top
richli
Posts: 6
Joined: Mon Sep 25, 2023 5:11 pm America/New_York
Answers: 0

Re: Missing checksums in VIIRS L3U products

by richli » Wed Mar 20, 2024 4:02 pm America/New_York

Thanks! I can confirm that all the files I listed have been fixed. However, I found additional files for 2024 with missing checksums:

20240105160000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240106112000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240110233000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240111085000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240115101000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240123042000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240125214000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240125210000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240130072000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240131104000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240201205000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240204155000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240205075000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240206075000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240216095000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240218092000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240221095000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240221102000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240221125000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240221152000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240223162000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240224095000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240224102000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240225152000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240225205000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240225212000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240227005000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240227015000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240227122000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240229025000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240301095000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240303012000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240303045000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240305194000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240318182000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240319012000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240319122000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240319182000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240320025000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240320065000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240320095000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240320075000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240320122000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
Top
PODAAC - wenhaoli
Subject Matter Expert
Subject Matter Expert
Posts: 35
Joined: Tue May 11, 2021 12:58 pm America/New_York
Answers: 0

Re: Missing checksums in VIIRS L3U products

by PODAAC - wenhaoli » Thu Mar 21, 2024 12:37 pm America/New_York

Thank you for discovering the missing md5 issue. PO.DAAC has just fixed the issue and re-ingested all the md5 files for the granules you have listed.
Top
richli
Posts: 6
Joined: Mon Sep 25, 2023 5:11 pm America/New_York
Answers: 0

Re: Missing checksums in VIIRS L3U products

by richli » Thu Mar 21, 2024 3:59 pm America/New_York

Thanks for the fix, everything looks good now!
Top
richli
Posts: 6
Joined: Mon Sep 25, 2023 5:11 pm America/New_York
Answers: 0

Re: Missing checksums in VIIRS L3U products

by richli » Fri Jun 14, 2024 8:51 pm America/New_York

The issue has recurred. I ran another check and the following files have missing checksums now:

20240203051000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240207015000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240208050000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240212223000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240214190000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240222162000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240225014000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240225021000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240228123000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240301025000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240301175000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240304153000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240309212000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240311075000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240314210000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240315224000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240317064000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240317184000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240322065000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240327062000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240402074000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240403224000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240405085000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240405051000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240408225000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240409072000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240409134000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240411063000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240419011000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240502101000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240518044000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240518224000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240521111000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240527234000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240530205000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240602075000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240605172000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240609152000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240609185000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240609172000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240611182000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc
20240611232000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240613182000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
20240614212000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
Top
viviant
Posts: 3
Joined: Fri Jun 21, 2024 3:32 pm America/New_York
Answers: 0

Re: Missing checksums in VIIRS L3U products

by viviant » Fri Jun 21, 2024 3:39 pm America/New_York

This could be due to a race condition with the discover granules process, which runs simultaneously with the data provider staging the md5 and .nc files. Our operation team will locate the data and md5 files and re-discover them for re-ingestion. Thank you for bringing this to our attention.
Top
PODAAC - celiaoued
Subject Matter Expert
Subject Matter Expert
Posts: 44
Joined: Fri May 28, 2021 1:30 pm America/New_York
Answers: 0

Re: Missing checksums in VIIRS L3U products

by PODAAC - celiaoued » Fri Jun 21, 2024 3:57 pm America/New_York

Thank you for alerting us to this recurrence. Our team will take corrective actions for the missing checksums.
Top
jakabasej8
Posts: 1
Joined: Sat Jun 22, 2024 3:58 am America/New_York
Answers: 0

Re: Missing checksums in VIIRS L3U products

by jakabasej8 » Sat Jun 22, 2024 3:59 am America/New_York

viviant wrote:
> This could be due to a race condition with the discover granules process,
> which runs simultaneously with the data provider staging the md5 and .nc
> files. Our operation team will locate the data and md5 files and
> re-discover them for re-ingestion. Thank you for bringing this to our
> attention.
Got it!
Top
Post Reply

Return to “Home”