l2bin doesn't work with OLCI and MSI

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
gbourdin
Posts: 34
Joined: Mon Jan 25, 2021 11:59 am America/New_York
Answers: 0
Has thanked: 1 time

l2bin doesn't work with OLCI and MSI

by gbourdin » Sun Oct 18, 2020 2:11 am America/New_York

Hello,
I am batch processing various L2 products using python.
My script works well with MODISA, MODIST and VIIRS L2 files either processed myself with l2gen or downloaded.
However it fails when I input OLCI and MSI images processed with l2gen with all default parameters.
I am using seadas-7.5.3  with python2.7 on CentOS-7 and I installed --olcis3a --olcis3b --msis2a and --msis2b since I managed to process L1 to L2 OLCI and MSI.
Note that l2bin and l3mapgen were working for OLCI until recently on the same system, but I recently tweaked many little details in my process to improve efficiency and now I can't manage to go back to where it was working.
The only difference between the process of L1 to L2 between MODIS/VIIRS and OLCI/MSI is that I did not use the land mask for OLCI/MSI and I used it for MODIS/VIIRS.

OLCI and MSI give two different errors.

My command to process OLCI with l2bin:
check_call([OCSSW_RUNNER, '--ocsswroot', PATH_OCSSW, 'l2bin',
    'ifile=/home/gbourdin/test/refl2bin_OC_test_OLCI_2016307.txt',
    'ofile=/home/gbourdin/test/2016307.L3b_CU_OC_test_OLCI_1km_1_DAY',
    'l3bprod=all',
    'prodtype=regional',
    'resolution=1',
    'flaguse=ATMFAIL,LAND,HILT,STRAYLIGHT,CHLFAIL',
    'night=False'])

Which gives:
Process L2bin ...
Loading command line parameters

INTERP parameter disabled.

1 input files
Averaging: standard
Resolution: 1
Max Qual Allowed: 255
prod_avg:
flagusemask: 33043
required: 0
NetCDF4 input file
/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.nc   brk:    0   4091 307  76691
total number of bins: 380187130
1970001 2038018 270
krow:     0 out of  17280  (-90.00 to -87.19) Sun Oct 18 00:49:07 2020
krow:   270 out of  17280  (-87.19 to -84.38) Sun Oct 18 00:49:07 2020
krow:   540 out of  17280  (-84.38 to -81.56) Sun Oct 18 00:49:07 2020
krow:   810 out of  17280  (-81.56 to -78.75) Sun Oct 18 00:49:07 2020
krow:  1080 out of  17280  (-78.75 to -75.94) Sun Oct 18 00:49:07 2020
krow:  1350 out of  17280  (-75.94 to -73.12) Sun Oct 18 00:49:07 2020
krow:  1620 out of  17280  (-73.12 to -70.31) Sun Oct 18 00:49:07 2020
krow:  1890 out of  17280  (-70.31 to -67.50) Sun Oct 18 00:49:07 2020
krow:  2160 out of  17280  (-67.50 to -64.69) Sun Oct 18 00:49:07 2020
krow:  2430 out of  17280  (-64.69 to -61.88) Sun Oct 18 00:49:07 2020
krow:  2700 out of  17280  (-61.88 to -59.06) Sun Oct 18 00:49:07 2020
krow:  2970 out of  17280  (-59.06 to -56.25) Sun Oct 18 00:49:07 2020
krow:  3240 out of  17280  (-56.25 to -53.44) Sun Oct 18 00:49:07 2020
krow:  3510 out of  17280  (-53.44 to -50.62) Sun Oct 18 00:49:07 2020
krow:  3780 out of  17280  (-50.62 to -47.81) Sun Oct 18 00:49:07 2020
krow:  4050 out of  17280  (-47.81 to -45.00) Sun Oct 18 00:49:07 2020
krow:  4320 out of  17280  (-45.00 to -42.19) Sun Oct 18 00:49:07 2020
krow:  4590 out of  17280  (-42.19 to -39.38) Sun Oct 18 00:49:07 2020
krow:  4860 out of  17280  (-39.38 to -36.56) Sun Oct 18 00:49:07 2020
krow:  5130 out of  17280  (-36.56 to -33.75) Sun Oct 18 00:49:07 2020
krow:  5400 out of  17280  (-33.75 to -30.94) Sun Oct 18 00:49:07 2020
krow:  5670 out of  17280  (-30.94 to -28.12) Sun Oct 18 00:49:07 2020
krow:  5940 out of  17280  (-28.12 to -25.31) Sun Oct 18 00:49:07 2020
krow:  6210 out of  17280  (-25.31 to -22.50) Sun Oct 18 00:49:07 2020
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 523: NetCDF: Not a valid ID for aot_865
Read Error: 0 (/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.ncû^O) 0
Traceback (most recent call last):
  File "processL3_unmerged.py", line 320, in <module>
    south_most=latmin, east_most=lonmax, west_most=lonmin, filename=filename, force=force_process)
  File "/home/gbourdin/tools.py", line 692, in process_L2_to_L3
    'night=' + night])
  File "/home/gbourdin/.conda/envs/SeaDAS/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/gbourdin/ocssw/scripts/ocssw_runner', '--ocsswroot', '/home/gbourdin/ocssw', 'l2bin', 'ifile=/home/gbourdin/test/refl2bin_OC_test_OLCI_2016307.txt', 'ofile=/home/gbourdin/test/2016307.L3b_CU_OC_test_OLCI_1km_1_DAY', 'l3bprod=all', 'prodtype=regional', 'resolution=1', 'flaguse=ATMFAIL,LAND,HILT,STRAYLIGHT,CHLFAIL', 'night=False']' returned non-zero exit status 255

I noticed this error that I never saw before:
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 523: NetCDF: Not a valid ID for aot_865
Read Error: 0 (/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.ncû^O) 0

So I checked all my L2 files to see if there were corrupted, but no, I can read them, and map them without any problem, including aot_865.

My command for MSI:
check_call([OCSSW_RUNNER, '--ocsswroot', PATH_OCSSW, 'l2bin',
    'ifile=/home/gbourdin/test/refl2bin_OC_test_MSI_2016308.txt',
    'ofile=/home/gbourdin/test/2016308.L3b_CU_OC_test_MSI_1km_1_DAY',
    'l3bprod=all',
    'prodtype=regional',
    'resolution=1',
    'flaguse=ATMFAIL,LAND,HILT,STRAYLIGHT,CHLFAIL',
    'night=False'])

Which gives:
Process L2bin ...
Loading command line parameters

INTERP parameter disabled.
8 input files
Averaging: standard
Resolution: 1
Max Qual Allowed: 2
prod_avg:
flagusemask: 33043
required: 0
NetCDF4 input file
-W- /home/seadas/ocssw/src/l2bin/l2bin.c line 1067: Unknown equator crossing time for sensorID=22 file=/home/gbourdin/test/S2A_MSIL1C_20161103T214052_N0204_R143_T02KME_20161103T214047.L2_SEN2.nc
Process L2bin ...
Traceback (most recent call last):
  File "processL3_unmerged.py", line 320, in <module>
    south_most=latmin, east_most=lonmax, west_most=lonmin, filename=filename, force=force_process)
  File "/home/gbourdin/process_OCfiles/L2_to_L3/tools.py", line 692, in process_L2_to_L3
    'night=' + night])
  File "/home/gbourdin/.conda/envs/SeaDAS/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/gbourdin/ocssw/scripts/ocssw_runner', '--ocsswroot', '/home/gbourdin/ocssw', 'l2bin', 'ifile=/home/gbourdin/test/refl2bin_OC_test_MSI_2016308.txt', 'ofile=/home/gbourdin/test/2016308.L3b_CU_OC_test_MSI_1km_1_DAY', 'l3bprod=all', 'prodtype=regional', 'resolution=1', 'flaguse=ATMFAIL,LAND,HILT,STRAYLIGHT,CHLFAIL', 'night=False']' returned non-zero exit status 1

Any thoughts?

Thank you,
Best,
Guillaume

Tags:

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

l2bin doesn't work with OLCI and MSI

by gnwiii » Sun Oct 18, 2020 8:13 am America/New_York

I noticed this error that I never saw before:
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 523: NetCDF: Not a valid ID for aot_865
Read Error: 0 (/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.ncû^O) 0
Looks like some edit left garbage ("û^O", looks like a "U" with a "hat" accent, then a "hat" followed by "O" here, but may be different or invisible elsewhere) at the end of a line.

gbourdin
Posts: 34
Joined: Mon Jan 25, 2021 11:59 am America/New_York
Answers: 0
Has thanked: 1 time

l2bin doesn't work with OLCI and MSI

by gbourdin » Sun Oct 18, 2020 6:31 pm America/New_York

Thank you for your prompt answer,
I looked for invisibles but could not find anything. I opened the input lists with various text editors, and looked for hidden symbols, but nothing.

I downloaded and processed again the same OLCI L1 to L2 but only keeping the two chlorophyll to see if it was aot_865 but I got the same error but for chlor_a with the same exit code:

-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 523: NetCDF: Not a valid ID for chlor_a
Read Error: 0 (/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.ncû^O) 0
attachment 1

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

l2bin doesn't work with OLCI and MSI

by gnwiii » Mon Oct 19, 2020 7:26 am America/New_York

Looking at the python code, you are passing the .txt file as the argument to "ifile=", so the error may be the netcdf library trying to open the .txt file.   You might find it better to create a parameter file for l2gen.

Post Reply