We are trying to set up automated downloads of PACE OCI BGC Level 2 files over our regions of interest. We set up a process to get products from this website: https://oceandata.sci.gsfc.nasa.gov/directdataaccess/Level-2/PACE-OCI and also set up downloads using the NASA Earthdata API.
My first question is if the file “Last Modified” timestamps are the actual arrival times of the files to these distribution servers? We have noted that the file “Last Modified” times are sometimes updated. Is there some kind of post processing than is done, or how can we know if the files are complete if the times are updated? And we also noted that the file availability are different based upon how we get the files, whether it is directly from the website or using the API. Can you help us to understand? The NRT labelled files can take 24+ hours to be completed based upon the file timestamps.
Thank you,
Kathy
PACI OCI Level 2 file NRT file latencies
-
OB.DAAC - SeanBailey
- User Services
- Posts: 64
- Joined: Mon Dec 16, 2024 8:43 am America/New_York
- Endorsed: 5 times
Re: PACI OCI Level 2 file NRT file latencies
Kathy,
Your question about the "Last Modified" times exposed an issue of sorts for the way that information is obtained. It's a long and boring story, so I'll save you the suffering and say that it's now fixed. The short answer is we store two times, the file create date and a "modify date". The former is what we should have been (and now are) reporting as the later is more for database informational changes.
If there is ever a question as to did the file actually change, you can use our "checkdata" API to retreive the sha1sum for the file and compare that to the shasum of the file you have obtained, e.g.:
https://oceandata.sci.gsfc.nasa.gov/checkdata/PACE_OCI.20250101T000242.L2.CLD.V3_0.NRT.nc
Now, using the directdataaccess page for your purpose is not ideal. It is better to set up a subscription (available from our Data Dashboard tool: https://oceandata.sci.gsfc.nasa.gov/data_dashboard). Unortunately, PACE hasn't been added to that yet, though we are actively working to include support for it. Earthdata Search also provides a subscription capability, though I don't think it would be easily scriptable.
BTW, the latency is typically on the order of 3-4 hours, though depending on a number of factors can be 12-24hrs, that should be rare.
Sean
Your question about the "Last Modified" times exposed an issue of sorts for the way that information is obtained. It's a long and boring story, so I'll save you the suffering and say that it's now fixed. The short answer is we store two times, the file create date and a "modify date". The former is what we should have been (and now are) reporting as the later is more for database informational changes.
If there is ever a question as to did the file actually change, you can use our "checkdata" API to retreive the sha1sum for the file and compare that to the shasum of the file you have obtained, e.g.:
https://oceandata.sci.gsfc.nasa.gov/checkdata/PACE_OCI.20250101T000242.L2.CLD.V3_0.NRT.nc
Now, using the directdataaccess page for your purpose is not ideal. It is better to set up a subscription (available from our Data Dashboard tool: https://oceandata.sci.gsfc.nasa.gov/data_dashboard). Unortunately, PACE hasn't been added to that yet, though we are actively working to include support for it. Earthdata Search also provides a subscription capability, though I don't think it would be easily scriptable.
BTW, the latency is typically on the order of 3-4 hours, though depending on a number of factors can be 12-24hrs, that should be rare.
Sean