An error when processing viirs Ibands with Seadas

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
zhouxingno1
Posts: 4
Joined: Thu Aug 11, 2022 5:09 pm America/New_York
Answers: 0

An error when processing viirs Ibands with Seadas

by zhouxingno1 » Mon Sep 19, 2022 11:58 am America/New_York

Hi,
I have installed Seadas 8.0 with ocssw V2022.0 in my system. I am able to generate level 2 M_band product from SNPP/JPSS VIIRS L1 product without any error. However I am getting below error while processing VIIRS I_bands data. Kindly suggest any appropriate solutions.

selecte I_band data generated by calibrate_viirs

selecte I_bands GEO data

set resolution=500

details
execution exception: java.io.IOException: l2gen failed with exit code 1.
Check log for more details.
Loading default parameters from /opt/SeaDAS/ocssw/share/common/msl12_defaults.par
Input file /mnt/Work/Algae/VIIRS/product/2022138/20221380400/SNPP_VIIRS.IMG.20220518T040000.L1B.hdf is a NetCDF4 VIIRS L1B file.

Loading characteristics for VIIRSN
Opening sensor information file /opt/SeaDAS/ocssw/share/viirs/npp/msl12_sensor_info.dat
Bnd Lam Fo Tau_r k_oz k_no2 t_co2 awhite aw bbw
0 410.000 171.134 3.175e-01 6.080e-04 5.951e-19 1.000e+00 1.000e+00 1.528e-01 2.931e-03
1 443.000 190.214 2.328e-01 3.349e-03 4.997e-19 1.000e+00 1.000e+00 3.484e-02 2.144e-03
2 486.000 198.774 1.600e-01 2.000e-02 2.990e-19 1.000e+00 1.000e+00 7.198e-02 1.447e-03
3 551.000 184.122 9.738e-02 8.937e-02 1.041e-19 1.000e+00 1.000e+00 9.721e-02 8.703e-04
4 671.000 150.456 4.395e-02 4.406e-02 1.015e-20 1.000e+00 8.871e-01 4.755e-01 3.842e-04
5 745.000 127.643 2.865e-02 1.111e-02 2.547e-21 1.000e+00 7.872e-01 2.806e+00 2.481e-04
6 862.000 96.031 1.594e-02 2.319e-03 4.358e-22 1.000e+00 6.486e-01 4.590e+00 1.361e-04
7 1238.000 45.606 3.650e-03 0.000e+00 0.000e+00 9.994e-01 0.000e+00 1.370e+02 3.044e-05
8 1601.000 25.084 1.305e-03 0.000e+00 0.000e+00 9.890e-01 0.000e+00 7.857e+02 1.070e-05
9 2257.000 7.732 3.294e-04 0.000e+00 0.000e+00 9.696e-01 0.000e+00 2.180e+03 2.656e-06

Bnd Lam
10 3700.000
11 4050.000
12 8550.000
13 10763.000
14 12013.000

Loading default parameters for VIIRSN from /opt/SeaDAS/ocssw/share/viirs/msl12_defaults.par
Loading default sub-sensor parameters for VIIRSN from /opt/SeaDAS/ocssw/share/viirs/npp/msl12_defaults.par
Loading parameters for suite OC from /opt/SeaDAS/ocssw/share/viirs/msl12_defaults_OC.par
Loading command line parameters

Loading user parameters for VIIRSN

Internal data compression requested at compression level: 4
Opening filter file /opt/SeaDAS/ocssw/share/viirs/msl12_filter.dat
Setting 25 x 7 straylight filter on HILT mask

Filter Kernel
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Minimum fill set to 1 pixels


Setting 25 x 7 straylight filter on CLDICE mask

Filter Kernel
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Minimum fill set to 1 pixels


Reading Thuillier_F0.dat
L1B Npix :31094976 Nscans:6496
Allocated 36939880 bytes in L1 record.
Allocated 14837760 bytes in L2 record.

Opening: /mnt/Work/Algae/VIIRS/product/2022138/20221380400/SNPP_VIIRS.20220518T040000.L2.OC.nc


The following products will be included in /mnt/Work/Algae/VIIRS/product/2022138/20221380400/SNPP_VIIRS.20220518T040000.L2.OC.nc.
0 rhos_671
1 rhos_862
2 l2_flags



Begin l2gen Version 9.5.1-V2022.0 Processing
Sensor is VIIRSN
Sensor ID is 11
Sensor has 10 reflective bands
Sensor has 5 emissive bands
Number of along-track detectors per band is 1
Number of input pixels per scan is 30912
Processing pixels 1 to 30912 by 1
Processing scans 1 to 6496 by 1
Ocean processing enabled
Land processing enabled
Atmospheric correction enabled

Begin MSl12 processing at 2022262233408000

Allocated 36939880 bytes in L1 record.
Allocated 36939880 bytes in L1 record.
Allocated 36939880 bytes in L1 record.
Allocated 36939880 bytes in L1 record.
Allocated 36939880 bytes in L1 record.
Allocated 36939880 bytes in L1 record.
Allocated 36939880 bytes in L1 record.
-E- /home/dshea/focs/oel_hdf4/libl1/l1_nc_generic_read.c line 142: nc_inq_ncid failed for file, /mnt/Work/Algae/VIIRS/product/2022138/20221380400/SNPP_VIIRS.IMG.20220518T040000.L1B.hdf group, geophysical_data.
file->nbands = 10

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: An error when processing viirs Ibands with Seadas

by OB.DAAC - SeanBailey » Mon Sep 19, 2022 12:48 pm America/New_York

That would be because we do not support I-band processing.

Sorry,
Sean

zhouxingno1
Posts: 4
Joined: Thu Aug 11, 2022 5:09 pm America/New_York
Answers: 0

Re: An error when processing viirs Ibands with Seadas

by zhouxingno1 » Wed Oct 05, 2022 6:13 am America/New_York

Thanks, and I want to process JPSS viirs data to get the I-band Rrc and Rrs product like VNP09, are there any software or simple ways to recommend?

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: An error when processing viirs Ibands with Seadas

by OB.DAAC - SeanBailey » Fri Oct 07, 2022 1:19 pm America/New_York

NOAA provides a surface reflectance product that includes the I-bands: https://www.ospo.noaa.gov/Products/land/sr/

jcoca
Posts: 20
Joined: Thu Feb 11, 2021 6:51 am America/New_York
Answers: 0

VIIRS-JPSS1 l2gen failure after R2022.14 tag instalation

by jcoca » Sat Oct 08, 2022 2:34 pm America/New_York

Hello

I have updated occsw to R2022.14 tag.
I am experiencing problems in processing ocean color data from VIIRS_JPSS1.

MODIS-AQUA, MODIS-TERRA, VIIRS_SNPP, OLCI-S3B worked OK from the beginnig. In case on S3A, minor changes were required. Changes in l2gen calls (wvl 682 to 681 and 866 to 865) and change the file ... /ocssw/share/olci/s3a/msl12_defaults_XM.par:

#
# MSL12 extraction and mapping Suite parameters for OLCI S3A
#

#l2prod=aot_866 angstrom Rrs_vvv chlor_a Kd_490 pic poc
l2prod=aot_865 angstrom Rrs_vvv chlor_a Kd_490 pic poc

However, in VIIRS_JPSS1 case, the processing fails when using l2gen (geolocate_viirs and calibrate_viirs works fine). I add a l2gen call and l2gen output for VIIRS_JPSS1:



l2gen call:

l2gen ifile=V2022271132400.L1B_JPSS1.nc geofile=/home/images/viirs_color/procesamiento/l1a/V2022271132400.GEO_JPSS1.nc ofile1=/home/images/viirs_color/procesamiento/l2/V2022271132400.L2_JPSS1.nc l2prod1=Kd_490,par,pic,poc,angstrom,aot_868,cdom_index,chlor_a,epsilon,Rrs_vvv,Rrs_746,Rrs_868,Rrs_1238,Rrs_1604,Rrs_2258,nLw_vvv,nLw_746,nLw_868,nLw_1238,nLw_1604,nLw_2258,rhos_vvv,rhos_746,rhos_868,rhos_1238,rhos_1604,rhos_2258,scattang,sena,senz,sola,solz,tindx_shi par=/home/images/viirs_color/procesamiento/l1b/V2022271132400.L1B_JPSS1.nc_corr.anc



l2gen output:

Loading default parameters from /seadas8/ocssw/share/common/msl12_defaults.par
Input file V2022271132400.L1B_JPSS1.nc is VIIRS NPP L1B NetCDF4.

Loading characteristics for VIIRSJ1
Opening sensor information file /seadas8/ocssw/share/viirs/j1/msl12_sensor_info.dat
Bnd Lam Fo Tau_r k_oz k_no2 t_co2 awhite aw bbw
0 411.000 172.571 3.210e-01 2.618e-04 6.006e-19 1.000e+00 1.000e+00 1.169e-02 2.984e-03
1 445.000 192.488 2.312e-01 3.220e-03 4.932e-19 1.000e+00 1.000e+00 1.833e-02 2.131e-03
2 489.000 197.681 1.573e-01 2.129e-02 2.886e-19 1.000e+00 1.000e+00 2.174e-02 1.428e-03
3 556.000 182.765 9.252e-02 9.965e-02 8.767e-20 1.000e+00 1.000e+00 7.367e-02 8.275e-04
4 667.000 151.106 4.420e-02 4.771e-02 8.008e-21 1.000e+00 8.871e-01 4.505e-01 3.883e-04
5 746.000 127.525 2.815e-02 1.083e-02 1.820e-21 1.000e+00 7.872e-01 2.833e+00 2.447e-04
6 868.000 94.945 1.533e-02 2.062e-03 6.223e-23 1.000e+00 6.486e-01 4.768e+00 1.315e-04
7 1238.000 45.609 3.650e-03 3.897e-14 0.000e+00 9.994e-01 0.000e+00 1.359e+02 3.044e-05
8 1604.000 24.974 1.296e-03 2.357e-14 0.000e+00 9.890e-01 0.000e+00 7.720e+02 1.063e-05
9 2258.000 7.711 3.285e-04 1.388e-13 0.000e+00 9.696e-01 0.000e+00 2.197e+03 2.648e-06

Bnd Lam
10 3700.000
11 4050.000
12 8550.000
13 10763.000
14 12013.000

Loading default parameters for VIIRSJ1 from /seadas8/ocssw/share/viirs/msl12_defaults.par
Loading default sub-sensor parameters for VIIRSJ1 from /seadas8/ocssw/share/viirs/j1/msl12_defaults.par
Loading parameters for suite OC from /seadas8/ocssw/share/viirs/msl12_defaults_OC.par
Loading parameters for suite OC from /seadas8/ocssw/share/viirs/j1/msl12_defaults_OC.par
Loading command line parameters

Loading user parameters for VIIRSJ1

Internal data compression requested at compression level: 4
Opening filter file /seadas8/ocssw/share/viirs/msl12_filter.dat
Setting 7 x 3 straylight filter on HILT mask

Filter Kernel
1 1 1 1 1 1 1
1 1 1 1 1 1 1
1 1 1 1 1 1 1

Minimum fill set to 1 pixels


Setting 7 x 3 straylight filter on CLDICE mask

Filter Kernel
1 1 1 1 1 1 1
1 1 1 1 1 1 1
1 1 1 1 1 1 1

Minimum fill set to 1 pixels


Reading Thuillier_F0.dat
Allocated 3824040 bytes in L1 record.
Allocated 1536000 bytes in L2 record.

Opening: /home/images/viirs_color/procesamiento/l2/V2022271132400.L2_JPSS1.nc


The following products will be included in /home/images/viirs_color/procesamiento/l2/V2022271132400.L2_JPSS1.nc.
0 Kd_490
1 par
2 pic
3 poc
4 angstrom
5 aot_868
6 cdom_index
7 chlor_a
8 epsilon
9 Rrs_411
10 Rrs_445
11 Rrs_489
12 Rrs_556
13 Rrs_667
14 Rrs_746
15 Rrs_868
16 Rrs_1238
17 Rrs_1604
18 Rrs_2258
19 nLw_411
20 nLw_445
21 nLw_489
22 nLw_556
23 nLw_667
24 nLw_746
25 nLw_868
26 nLw_1238
27 nLw_1604
28 nLw_2258
29 rhos_411
30 rhos_445
31 rhos_489
32 rhos_556
33 rhos_667
34 rhos_746
35 rhos_868
36 rhos_1238
37 rhos_1604
38 rhos_2258
39 scattang
40 sena
41 senz
42 sola
43 solz
44 tindx_shi
45 l2_flags



Begin l2gen Version 9.6.0-R2022.14 Processing
Sensor is VIIRSJ1
Sensor ID is 25
Sensor has 10 reflective bands
Sensor has 5 emissive bands
Number of along-track detectors per band is 16
Number of input pixels per scan is 3200
Processing pixels 1 to 3200 by 1
Processing scans 1 to 3232 by 1
Ocean processing enabled
Land processing enabled
Atmospheric correction enabled

Begin MSl12 processing at 2022280113916000

Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Loading radiance to brightness temperature from /seadas8/ocssw/share/viirs/npp/cal/bt_viirsn.hdf
Loading land mask information from /seadas8/ocssw/share/common/gebco_ocssw_v2020.nc
Loading DEM information from /seadas8/ocssw/share/common/gebco_ocssw_v2020.nc
Loading ice mask file from /seadas8/ocssw/var/anc/2022/271/N202227100_SEAICE_NSIDC_24h.hdf
Loaded near real time NSIDC ice HDF file.
Loading XCAL rvs and polarization sensitivities from _411.hdf
-E- /home/dshea/focs/oel_hdf4/libl1/xcal.c line 104: SDstart(_411.hdf, 1) failed.

The directory for VIIRS-SNPP ..../ocssw/share/viirs/npp/cal/
contains the polcor_viirsn_*.hdf and rsvcor_viirsnpp_*.hdf files

However the directory .../ocssw/share/viirs/j1/cal/
only contains the polcor_viirsj1_*.hdf files, not the rscor*.hdf files

I do not know if that could be the problem, as the V2022.0 tag .../ocssw/share/viirs/j1/cal/ directory neither contained the rvscor*.hdf files.

I have reinstaled the --viirsj1 processing files, deleting the previous --.../ocssw/share/viirs/j1/cal/ directory. I tried to use the files from a V2022.0 tag instalation. However the l2gen failure output messages wee the same.

Best regards
Josep

jcoca
Posts: 20
Joined: Thu Feb 11, 2021 6:51 am America/New_York
Answers: 0

Re: VIIRS-JPSS1 l2gen failure after R2022.14 tag instalation

by jcoca » Tue Oct 11, 2022 11:29 am America/New_York

Hello

I regularly run update_luts. The last files of for jpss1 in the directory .../ocssw/var/viirsj1/cal/OPER/ correspond to 29th September 2022:

VIIRS_J1_CAL_DNB_DYNAMIC_LUT_20180106_20230422_v3.1.1.3_20220907_obpg.nc
VIIRS_J1_CAL_RSB_DYNAMIC_LUT_20171213_20240101_v3.1.1.3_20220907_obpg.nc
VIIRS_J1_CAL_STATIC_LUT_v3.1.1.3_20220808_vcst.nc
VIIRS_J1_CAL_STRAYLIGHT_DYNAMIC_LUT_20180117_20230419_v3.1.1.3_20220907_obpg.nc
VIIRS_J1_CMN_LUT_v3.1.1.3_20220808_vcst.nc
VIIRS_J1_GEO_LUT_v3.1.0.1_20220510_vcst.nc

I have deleted all the files in the direotory and run update_luts viirsn again. Just in case of some posible corruption (I have experienced that in some ocasion with ancillary data).However the problem remains

Regards
Josep

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: VIIRS-JPSS1 l2gen failure after R2022.14 tag instalation

by OB.DAAC - SeanBailey » Tue Oct 11, 2022 1:34 pm America/New_York

Josep,
Seems we missed a required modification to the LutUtils.py code. For the previous reprocessing, JPSS1_VIIRS did not require an xcal file, so LutUtils.py didn't look for them to pull down. Until we can get the change into a tag, you can make the following change to $OCSSW_BIN/seadasutils/LutUtils.py:

diff --git a/src/scripts/seadasutils/LutUtils.py b/src/scripts/seadasutils/LutUtils.py
index f6919c37..19a14aea 100644
--- a/src/scripts/seadasutils/LutUtils.py
+++ b/src/scripts/seadasutils/LutUtils.py
@@ -90,10 +90,7 @@ class LutUtils:

# add calibration dirs for MODIS and VIIRS
if self.sensor['instrument'] in ['MODIS', 'VIIRS']:
- if sensor == 'viirsj1':
- caldirs = ['cal']
- else:
- caldirs = ['cal', 'xcal']
+ caldirs = ['cal', 'xcal']
for caldir in caldirs:
dirs.append(os.path.join(sensor, caldir, 'OPER'))
if self.evalluts:

...or just make line 96 look like this:
caldirs = ['cal', 'xcal']

Sean

jcoca
Posts: 20
Joined: Thu Feb 11, 2021 6:51 am America/New_York
Answers: 0

Re: VIIRS-JPSS1 l2gen failure after R2022.14 tag instalation

by jcoca » Wed Oct 12, 2022 4:27 pm America/New_York

Hello Thank you very much, with this minor change it is working now.

I was wondering if it could be related to some issue related to xcal files, as noticed that in previous tags were not required for jpss1

Regards
Josep

zhouxingno1
Posts: 4
Joined: Thu Aug 11, 2022 5:09 pm America/New_York
Answers: 0

How to get Rrc from VIIRS I-bands

by zhouxingno1 » Wed Oct 19, 2022 12:30 am America/New_York

Hello
I'm interested in getting Rayleigh-corrected reflectance from JPSS&NPP VIIRS I-band (VNP02IMG & VJ102MOD data), but I'm having trouble figuring out how to use Rayleigh-corrected method to correct it. And I tried VNP09 data,but I can't find the same products from JPSS-VIIRS. Any help is much appreciated, thank you!

Post Reply