The new file naming convention introduces a .NRT prefix for Near Real Time products. How is it related to previous QLP, QL and RF consolidation flavors? Does the .NRT suffix include QLP and QL products?
With previous reprocessings we used the processing_version NetCDF attribute to differentiate QLP, QL and RF products (for example it contained respectively "2018.1QLP", "2018.1QL" or "2018.1").
How could we differentiate them with the new MODIS reprocessing, because now the processing_version attribute seems always contain "R2022.0"? Is searching the presence of ".NRT." in the input_sources attribute a good solution?
For MODIS Aqua QL products were generated in general 16 days after the QLP version, is it the same logic with the new R2022 reprocessing?
As you noted, with the R2022 reprocessing we are including ".NRT" in the filename for, well, near real-time data. This is the equivalent of the previous QL (and QLP) designation. There will only be one "quicklook" (NRT) product produced prior to the final (is it ever final?) refined product, so no more QLP. I'll have to check with our production folks as to why the version attribute doesn't have the 'QL' moniker...it probably should filenames can lie (I thought it would've been in there... I might ask them to make use NRT instead of QL)
Might be a week or so, though, as the person I'd ask is on vacation (and if he's reading this before his scheduled return to work - he should turn around and head back to the pool deck with a cold beverage in hand...)
I was also in vacation last week and I can confirm that read work emails during vacations is not a good idea for vacation health!
I'm a bit surprised that there is no more QLP with the R2022 Aqua reprocessing. With previous Aqua reprocessing we have typically a QLP generated first, then replaced by a QL 16 days after, then replaced by a RF later. What is the logic with R2022? By looking at some R2022 products metadata it seems that Aqua files are still regenerated 16 days after then another time for RF, am I wrong? On our side for the moment we monitor QLP, QL and RF flux separately to be able to detect potential suspect delay on each flux. I am also interested to know what will be the logic for VIIRS R2022, do we still have QLP?
I found also this interesting explanation below from you in 2018 about the subject. I understand that VIIRS RF == Aqua QL in term of product consolidation/refinement (I confirm that they are both generated 16 days later), will this logic be the same with R2022?
JulienFor VIIRS, we receive the Level 0 data twice - once as a session-based file (i.e. ground station contact) and later as a time-based file (replayed as exact 2-hr chunks). The session-based files receive the QLP designation and once the time-based version is available, it gets the QL designation.
When we implemented the revised calibration update procedure for MODIS-Aqua, we defined QLP for MODIS to be data generated with the as-available ancillary data. Once the final coincident ancillary data are available, the data are reprocessed and considered QL until the revised calibration is available at which point the refined reprocessing is done. So, for MODIS, the current QL is the equivalent to the previous definition of refined, while QLP is equivalent to the previous QL definition.
I was reminded by the production staff that with the filenames now having NRT in them, the confusion around quicklook and refined was eliminated, so the version element in the metadata would no longer need the QL[P] modifier. Seems that clarification caused confusion
Prior to having a version element in our filenames (NRT is a "version" - a future reprocessing may put a version in the non-NRT data as well...), we needed a way to identify the quicklook data from the refined data (and an undesirable situation with the ancillary data gave birth to the QLP concept). Thus the "QL[P]" modifier to the processing version metadata. With R2022, NRT is in the filename *and* in the product_name attribute (e.g. :product_name = "AQUA_MODIS.20220804T162501.L2.OC.NRT.nc") - so even though filenames can lie, the metadata still hangs on to it's honesty.
With R2022, there are only NRT and refined - with NRT==QL. QLP is no longer necessary. We're using a different ancillary data source that has a "refined" cadence that matches (well is close to) our forward stream calibration update cadence (the same frequency as it was prior to R2022). Files will remain NRT until the calibration/ancillary refined processing is done.
It hasn't been a month since we reprocessed Aqua-MODIS (or a month since we put the R2022 configuration in the forward stream), so we're a bit out of step with our nominal QL/RF cadence. In fact, the R2022 RF is currently through Feb 2022. Data from March to the present are NRT. We are working on the update to the calibration to bring the R2022 RF processing up to date now.
This same process will be used for VIIRS when it is reprocessed to as R2022, including the forward stream calibration updates (which were not previously applied to VIIRS).
In summary, for R2022:
1) Only TWO statuses: NRT and refined
2) Files will be reprocessed as refined only after the MERRA2 ancillary data are available AND the forward-stream calibration is ready.
3) MODIS and VIIRS will both follow the same NRT/RF logic
The new .NRT suffix allows to identify quickly NRT files, which is fine, and having now only NRT and RF is less confusing than RF/QL/QLP I think.
The only drawback is that for the moment we have to handle both the new logic for Aqua and the old logic for VIIRS.
Is there already a schedule for reprocessing of both VIIRS?
Hopefully, Terra-MODIS will be done by the end of August, early Sept for SNPP-VIIRS and late Sept for NOAA20-VIIRS...just don't hold me to those dates