Processing version

[khyde]

Hello,

I use l2gen (SeaDAS 7.4) to create most of my L2 files, but it is unclear how to make sure I am generating R2018 files. I routinely run update_luts and have recently updated the MODISA and VIIRS OC Processors from within SeaDAS, but I don't think my latest files are actually R2018.  When looking at the L2 attributes, the processing_version is listed as "unspecified" and the doi is 10.5067/AQUA/MODIS_OC.2014.0.  If I compare that to L2 files downloaded directly from the website, the processing version is "2018.0QLP" and the doi is 10.5067/AQUA/MODIS/L2/OC/2018.  Is there something else I need to update to make the R2018 files?

Also, besides running Update OC Processors, is there an easy way from within SeaDAS to tell if the OCCSW processing code for each sensor is current (i.e. R2018)?  I am just looking for tips on how to make sure the L2 files I generate are using the latest processing code and calibration files so that they match your L2 files. 

Thanks,
Kim

[gnwiii]

There is quite a lot of detail in the level-2 netcdf "headers", which you can view using netcdf -h <l2file>.  I assume you are creating level-2 products that NASA doesn't provide, but you can also do band-by-band comparisons of an R2018.0 file with test files processed locally using the same options.  I've done a bit of this and found a change in the details of the seasonal ice mask (a minor thing from a global viewpoint, but signficant for Alaska and Canada).  Key things to check in the headers are calibration and ancillary files. 

I assume you have seen the posts noting that you need to manually edit msl12_defaults.par to update the vicarious calibration and names of the files downloaded by update_luts.py.  You can also check the output when you run l2gen for the "expected" ancillary and LUT files.

[OB.DAAC - SeanBailey]

For MODIS-Aqua, as long as you have the latest LUTs and use the gains listed on the R2018 reprocessing page, the files you create will match those we create (well, that and if you use the same ancillary inputs).

For VIIRS, the same is true with the caveat that the LUT information needs to be put in the $OCDATAROOT/viirsn/instrument_defaults.par file (or an equivalent file that is passed to l2gen using the viirscalparfile input option)...

Sean

[khyde]

Thanks Sean,

I compared the gains in the msl12_defaults.par file and they hadn't been updated either so I fixed that manually.  At what step does the msl12_defaults file get updated (or is supposed to be updated)?  Is it when you run update_luts or just when new xcal files are downloaded?  I am just trying to figure out how frequently I need to check the msl12 file to make sure it is current.

I am also curious about the global attributes in the files and how to get the correct processing version.  It seems like checking the processing version would be the easiest way to determine if the files are R2018, R2018QL or R2018QLP, but right now my files all have "unspecified" in the processing_version tag.  Before, it was easy to search the attributes or log files and compare the ancillary files to know if the file needed to be recreated with updated ancillary data, but with the delayed calibration files it doesn't seem like there is an easy way to know if the files I create are "refined" or will need to be reprocessed with the next update.

Thanks again,
Kim

[OB.DAAC - SeanBailey]

Kim,

Generally, the defaults file will not change. The xcal file update does (at least in v7.4 and earlier) modify the defaults file, but only updates the xcalfile parameter.  The gains do not change between reprocessings.  Normally our SeaDAS release is tied a bit closer to the reprocessing events, and the new release would have the necessary modifications to the default files.  This release is taking a wee bit longer, and so you find yourself in the situation your in :sad:

With the next release, the xcalfile parameter will also not need to change in the defaults file - the code was modified to identify what version to use by what version is in the OPER directory.
So, soon the defaults file can remain untouched.

The processing version attribute will remain unspecified (unless you specify it - but there's no need for you to do so).  It is set by the pversion input parameter.  We set it so users who grab our data will know if it is QL or refined and what reprocessing it was generated under.  For MODIS, the calibration files used are reported in the calibration_data attribute, e.g.:

$ ncdump -h A2018015000500.L2_new.nc|grep -i calibration_data
                :calibration_data = "A2018015000500.L1A_LAC, MYD02_Reflective_LUTs.V6.2.1.6_OC_v1.5.hdf, MYD02_Emissive_LUTs.V6.2.1.6_OC_v1.5.hdf, MYD02_QA_LUTs.V6.2.1.6_OC_v1.5.hdf" ;

VIIRS is buried deeper and so didn't bubble up to get reported there...I'll see about fixing that :grin:

Sean

[khyde]

Thanks for the explanation Sean,

For now I will keep doing what I am doing until the next release of SeaDAS.

As for the calibration files, how do you know which files should be updated when new calibration files are released?  For example, the latest files have a time stamp of Feb 13th.  How far back do I need to go to make my files "refined"?  Would 60 or 90 days be safe?

I will hold off on VIIRS until after the next release of SeaDAS.

Thanks again,
Kim

[OB.DAAC - SeanBailey]

Well, you can't know....at least not easily.  For MODIS-Aqua, we're updating the calibration approximately monthly, and we've arbitrarily chosen the 1st of the previous month for when the new lut is valid through.  For example, the February update (created on 14 Feb) allows data through 31 December to be reprocessed as refined. I've been mulling around an idea to have a status page for this kind of information, but haven't had time to implement it.

Sean

[khyde]

Thank Sean,
I can work with that info for now.
Kim