What is the processing flow for VIIRS for L0 -> L1 -> L2 -> l2bin -> l3mapgen

[woodbri]

Hi All,

If the process in SeaDAS is documented for VIIRS, please point me to that, otherwise can someone outline the best processing flow for VIIRSnpp and VIIRSj1 to get from L0 -> ... -> l3mapgen?

I looking to extract hires SST, OC, and true color geotiff files in the final step.

I have the MODIS work flow working, but that depends on a bunch of modis_*.py scripts for a lot of that processing. I'm fine with scripting up a new workflow but I'm not sure and what steps I need to follow. In modis processing we call getanc.py, modis_atteph.py, what needs to happen in the VIIRS processing?

I see the following viirs specific files in bin:
bin/calibrate_viirs
bin/geolocate_viirs
bin/l1aextract_viirs
bin/l2bin_viirs
bin/viirs_obc

For example if I start with these:
V2020078180000.L1A_SNPP.nc
V2020078185400.L1A_JPSS1.nc

Thanks!

[OB SeaDAS - dshea]

If you are grabbing the L1A files from us, you might as well get the GEO files from us also also.  Dealing with the L0 files is more difficult, and we do not distribute the L0->L1A program.

L1A:
V2020078180000.L1A_SNPP.nc
V2020078185400.L1A_JPSS1.nc

GEO:
V2020078180000.GEO-M_SNPP.nc 
V2020078185400.GEO-M_JPSS1.nc

L2:
l2gen ifile=V2020078180000.L1A_SNPP.nc geofile=V2020078180000.GEO-M_SNPP.nc ofile=V2020078180000.L2_SNPP_OC.nc <other params>

Now that you have an L2 file everything is the same as MODIS.  You could possibly play with the resolution of the binning, but all processing from L2 onward is similar.

don

[OB.DAAC - SeanBailey]

As Don subtly mentions, if you're using full L1A files, you don't need to run geolocate_viirs - just grab the one we distribute.
If you have an extracted L1A VIIRS file, then you'd need to run geolocate_viirs.  It's pretty easy:
$ geolocate_viirs ifile=<L1Afile> geofile_mod=<filebase.GEO-M.nc>
(the geolocation code can spit out geolocation files for 3 resolutions, moderate, imaging and DNB (day/night band).  You only need the "mod")

Likewise, calibrate_viirs isn't necessary as input to l2gen, as it will be calibrated when l2gen reads the L1A file.

Once you have an L2 file, the rest of the flow is identical to what you're doing for MODIS.

Sean

[woodbri]

Thanks, both your responses were helpful and I have been able to get Chlor_a and Kd_490 output, but things blew up on the sst and true color with the following:

...
Loading SSES table from /u/oceandata/share/viirs/npp/cal/sst_sses_viirsn.v6.4.1.gt100counts.hdf
5,6,4,4,4,2,7
-E- /home/seadas/ocssw/src/l2gen/sst.c line 869: Table dimensions for bias do not match expectation.

Failed: 'l2gen ifile=V2020079165400.L1A_JPSS1.nc geofile=V2020079165400.GEO-M_JPSS1.nc ofile=V2020079165400.sst.L2.nc l2prod=sst,rhos_vvv par=V2020079165400.L1A_JPSS1.nc.anc resolution=250', error 256

and probably because the above failed:

l2bin prodtype=regional ofile=V2020079165400.sst.L3b.nc infile=V2020079165400.sst.L2.nc resolution=Q average=area l3bprod=sst
l2bin 5.0.0 (Aug 16 2019 12:47:25)
Loading default parameters from /u/oceandata/share/viirs/l2bin_defaults.par
Loading command line parameters

INTERP parameter disabled.
-E- /home/seadas/ocssw/src/libl2/readL2scan.c:404 - openL2 - could not find time_coverage_start.

Failed: 'l2bin prodtype=regional ofile=V2020079165400.sst.L3b.nc infile=V2020079165400.sst.L2.nc resolution=Q average=area l3bprod=sst', error 256
l2bin suite=SFREFL ifile=V2020079165400.sst.L2.nc ofile=V2020079165400.tc.L3b.nc resolution=Q average=area l3bprod=rhos_469,rhos_555,rhos_645
l2bin 5.0.0 (Aug 16 2019 12:47:25)
Loading default parameters from /u/oceandata/share/viirs/l2bin_defaults.par
Loading default parameters from /u/oceandata/share/viirs/l2bin_defaults_SFREFL.par
Loading command line parameters

INTERP parameter disabled.
-E- /home/seadas/ocssw/src/libl2/readL2scan.c:404 - openL2 - could not find time_coverage_start.

Failed: 'l2bin suite=SFREFL ifile=V2020079165400.sst.L2.nc ofile=V2020079165400.tc.L3b.nc resolution=Q average=area l3bprod=rhos_469,rhos_555,rhos_645', error 256

[OB.DAAC - SeanBailey]

You need to make sure your viirs data trees are up-to-date.  rerun install_ocssw.py with --viirsn --viirsj1

[woodbri]

I reinstalled ocssw with:

mkdir -p /u/oceandata
cd /u/oceandata

wget -N https://oceandata.sci.gsfc.nasa.gov/ocssw/install_ocssw.py
chmod +x install_ocssw.py

./install_ocssw.py -v --install-dir=/u/oceandata  --git-branch=v7.5 --arch=linux_64 --aqua --avhrr --terra --viirsn --viirsj1 --viirsdem --src

but I'm getting the same errors with SST FAILING, and if this helps:

$ ls -l /u/oceandata/share/viirs/npp/cal/sst_sses_viirsn.v6.4.1.gt100counts.hdf
-rw-r--r-- 1 root root 384766 Feb 29 13:28 /u/oceandata/share/viirs/npp/cal/sst_sses_viirsn.v6.4.1.gt100counts.hdf

Also it looks like I need to adjust true color processing based on:

The following products will be included in V2020079165400.sst.L2.nc.
0 sst
1 rhos_411
2 rhos_445
3 rhos_489
4 rhos_556
5 rhos_667
6 l2_flags

with something like l3bprod=rhos_489,rhos_556,rhos_667 since the the sensors seem to be slightly different from modis.

[woodbri]

I have gotten the true color to work by running l2gen with just the rhos_vvv product listed, (ie: don't include sst) and then binning that.

BUT I have not figured out the problem with the SST processing:

l2gen ifile=V2020079165400.L1A_JPSS1.nc geofile=V2020079165400.GEO-M_JPSS1.nc ofile=V2020079165400.sst.L2.nc l2prod=sst par=V2020079165400.L1A_JPSS1.nc.anc resolution=375

[snip lots of output]

Applying ocean BRDF including:
    Reflection/refraction for upwelling radiance.
    Reflection/refraction for downwelling radiance.
    Morel f/Q

Raman correction unsupported for this sensor.
No Raman scattering correction calculated for Rrs.

Loading SSES table from /u/oceandata/share/viirs/npp/cal/sst_sses_viirsn.v6.4.1.gt100counts.hdf
5,6,4,4,4,2,7
-E- /home/seadas/ocssw/src/l2gen/sst.c line 869: Table dimensions for bias do not match expectation.

Failed: 'l2gen ifile=V2020079165400.L1A_JPSS1.nc geofile=V2020079165400.GEO-M_JPSS1.nc ofile=V2020079165400.sst.L2.nc l2prod=sst par=V2020079165400.L1A_JPSS1.nc.anc resolution=375', error 256

[OB.DAAC - SeanBailey]

A couple of things to point out...

1) the resolution option for l2gen only applies to MODIS.  While VIIRS does have "I-bands" at 375m resolution, l2gen only supports the "M-bands".

2) While VIIRS on JPSS-1 (NOAA-20) is capable of SST, l2gen only supports SST for VIIRS on SNPP.  It's a bit of an issue, to be sure.  We *could* support it, but there currently is no funded PI to be responsible for it, and thus we do not have the necessary coefficient tables for it.   It is referred to as an orphaned product. 

I realize now I should have put the SST coefficient and SSES bias table information is the viirs/npp defaults file not at the root viirs sensor default file.  Had I done that it would have been more clear:
Processing scan #     0 (1 of 3232) after      2 seconds
No SSES data provided for this sensor..

I'll fix that in the next release.

Sean

[woodbri]

Thank you for the update.
So I'm clear on this:

1. resolution is ignored for viirs in l2gen, and on l2bin and l3mapgen they still determine the output resolution of the image, but that just spreads the coarser M-band data over more pixels.

2. at the moment neither of the VIIRS support l2gen for SST, until the next release when SNPP will after you update the defaults.

Thanks again for your assistance.
-Steve

[OB.DAAC - SeanBailey]

1) yes
2) no

SST is functional and supported for VIIRS-SNPP, just not VIIRS-JPSS1.  The change that I will make will move the definition of the SSES files to the npp default file, so when jpss1 is processed it will not have any files listed and will appropriately complain with an error message that is more sensible than the one you were seeing.

Sean