Hello,
I am batch processing various L2 products using python.
My script works well with MODISA, MODIST and VIIRS L2 files either processed myself with l2gen or downloaded.
However it fails when I input OLCI and MSI images processed with l2gen with all default parameters.
I am using seadas-7.5.3 with python2.7 on CentOS-7 and I installed --olcis3a --olcis3b --msis2a and --msis2b since I managed to process L1 to L2 OLCI and MSI.
Note that l2bin and l3mapgen were working for OLCI until recently on the same system, but I recently tweaked many little details in my process to improve efficiency and now I can't manage to go back to where it was working.
The only difference between the process of L1 to L2 between MODIS/VIIRS and OLCI/MSI is that I did not use the land mask for OLCI/MSI and I used it for MODIS/VIIRS.
OLCI and MSI give two different errors.
My command to process OLCI with l2bin:
check_call([OCSSW_RUNNER, '--ocsswroot', PATH_OCSSW, 'l2bin',
'ifile=/home/gbourdin/test/refl2bin_OC_test_OLCI_2016307.txt',
'ofile=/home/gbourdin/test/2016307.L3b_CU_OC_test_OLCI_1km_1_DAY',
'l3bprod=all',
'prodtype=regional',
'resolution=1',
'flaguse=ATMFAIL,LAND,HILT,STRAYLIGHT,CHLFAIL',
'night=False'])
Which gives:
Process L2bin ...
Loading command line parameters
INTERP parameter disabled.
1 input files
Averaging: standard
Resolution: 1
Max Qual Allowed: 255
prod_avg:
flagusemask: 33043
required: 0
NetCDF4 input file
/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.nc brk: 0 4091 307 76691
total number of bins: 380187130
1970001 2038018 270
krow: 0 out of 17280 (-90.00 to -87.19) Sun Oct 18 00:49:07 2020
krow: 270 out of 17280 (-87.19 to -84.38) Sun Oct 18 00:49:07 2020
krow: 540 out of 17280 (-84.38 to -81.56) Sun Oct 18 00:49:07 2020
krow: 810 out of 17280 (-81.56 to -78.75) Sun Oct 18 00:49:07 2020
krow: 1080 out of 17280 (-78.75 to -75.94) Sun Oct 18 00:49:07 2020
krow: 1350 out of 17280 (-75.94 to -73.12) Sun Oct 18 00:49:07 2020
krow: 1620 out of 17280 (-73.12 to -70.31) Sun Oct 18 00:49:07 2020
krow: 1890 out of 17280 (-70.31 to -67.50) Sun Oct 18 00:49:07 2020
krow: 2160 out of 17280 (-67.50 to -64.69) Sun Oct 18 00:49:07 2020
krow: 2430 out of 17280 (-64.69 to -61.88) Sun Oct 18 00:49:07 2020
krow: 2700 out of 17280 (-61.88 to -59.06) Sun Oct 18 00:49:07 2020
krow: 2970 out of 17280 (-59.06 to -56.25) Sun Oct 18 00:49:07 2020
krow: 3240 out of 17280 (-56.25 to -53.44) Sun Oct 18 00:49:07 2020
krow: 3510 out of 17280 (-53.44 to -50.62) Sun Oct 18 00:49:07 2020
krow: 3780 out of 17280 (-50.62 to -47.81) Sun Oct 18 00:49:07 2020
krow: 4050 out of 17280 (-47.81 to -45.00) Sun Oct 18 00:49:07 2020
krow: 4320 out of 17280 (-45.00 to -42.19) Sun Oct 18 00:49:07 2020
krow: 4590 out of 17280 (-42.19 to -39.38) Sun Oct 18 00:49:07 2020
krow: 4860 out of 17280 (-39.38 to -36.56) Sun Oct 18 00:49:07 2020
krow: 5130 out of 17280 (-36.56 to -33.75) Sun Oct 18 00:49:07 2020
krow: 5400 out of 17280 (-33.75 to -30.94) Sun Oct 18 00:49:07 2020
krow: 5670 out of 17280 (-30.94 to -28.12) Sun Oct 18 00:49:07 2020
krow: 5940 out of 17280 (-28.12 to -25.31) Sun Oct 18 00:49:07 2020
krow: 6210 out of 17280 (-25.31 to -22.50) Sun Oct 18 00:49:07 2020
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 523: NetCDF: Not a valid ID for aot_865
Read Error: 0 (/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.ncû^O) 0
Traceback (most recent call last):
File "processL3_unmerged.py", line 320, in <module>
south_most=latmin, east_most=lonmax, west_most=lonmin, filename=filename, force=force_process)
File "/home/gbourdin/tools.py", line 692, in process_L2_to_L3
'night=' + night])
File "/home/gbourdin/.conda/envs/SeaDAS/lib/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/gbourdin/ocssw/scripts/ocssw_runner', '--ocsswroot', '/home/gbourdin/ocssw', 'l2bin', 'ifile=/home/gbourdin/test/refl2bin_OC_test_OLCI_2016307.txt', 'ofile=/home/gbourdin/test/2016307.L3b_CU_OC_test_OLCI_1km_1_DAY', 'l3bprod=all', 'prodtype=regional', 'resolution=1', 'flaguse=ATMFAIL,LAND,HILT,STRAYLIGHT,CHLFAIL', 'night=False']' returned non-zero exit status 255
I noticed this error that I never saw before:
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 523: NetCDF: Not a valid ID for aot_865
Read Error: 0 (/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.ncû^O) 0
So I checked all my L2 files to see if there were corrupted, but no, I can read them, and map them without any problem, including aot_865.
My command for MSI:
check_call([OCSSW_RUNNER, '--ocsswroot', PATH_OCSSW, 'l2bin',
'ifile=/home/gbourdin/test/refl2bin_OC_test_MSI_2016308.txt',
'ofile=/home/gbourdin/test/2016308.L3b_CU_OC_test_MSI_1km_1_DAY',
'l3bprod=all',
'prodtype=regional',
'resolution=1',
'flaguse=ATMFAIL,LAND,HILT,STRAYLIGHT,CHLFAIL',
'night=False'])
Which gives:
Process L2bin ...
Loading command line parameters
INTERP parameter disabled.
8 input files
Averaging: standard
Resolution: 1
Max Qual Allowed: 2
prod_avg:
flagusemask: 33043
required: 0
NetCDF4 input file
-W- /home/seadas/ocssw/src/l2bin/l2bin.c line 1067: Unknown equator crossing time for sensorID=22 file=/home/gbourdin/test/S2A_MSIL1C_20161103T214052_N0204_R143_T02KME_20161103T214047.L2_SEN2.nc
Process L2bin ...
Traceback (most recent call last):
File "processL3_unmerged.py", line 320, in <module>
south_most=latmin, east_most=lonmax, west_most=lonmin, filename=filename, force=force_process)
File "/home/gbourdin/process_OCfiles/L2_to_L3/tools.py", line 692, in process_L2_to_L3
'night=' + night])
File "/home/gbourdin/.conda/envs/SeaDAS/lib/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/gbourdin/ocssw/scripts/ocssw_runner', '--ocsswroot', '/home/gbourdin/ocssw', 'l2bin', 'ifile=/home/gbourdin/test/refl2bin_OC_test_MSI_2016308.txt', 'ofile=/home/gbourdin/test/2016308.L3b_CU_OC_test_MSI_1km_1_DAY', 'l3bprod=all', 'prodtype=regional', 'resolution=1', 'flaguse=ATMFAIL,LAND,HILT,STRAYLIGHT,CHLFAIL', 'night=False']' returned non-zero exit status 1
Any thoughts?
Thank you,
Best,
Guillaume
l2bin doesn't work with OLCI and MSI
l2bin doesn't work with OLCI and MSI
I noticed this error that I never saw before:
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 523: NetCDF: Not a valid ID for aot_865
Read Error: 0 (/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.ncû^O) 0
Looks like some edit left garbage ("û^O
", looks like a "U" with a "hat" accent, then a "hat" followed by "O" here, but may be different or invisible elsewhere) at the end of a line.l2bin doesn't work with OLCI and MSI
Thank you for your prompt answer,
I looked for invisibles but could not find anything. I opened the input lists with various text editors, and looked for hidden symbols, but nothing.
I downloaded and processed again the same OLCI L1 to L2 but only keeping the two chlorophyll to see if it was aot_865 but I got the same error but for chlor_a with the same exit code:
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 523: NetCDF: Not a valid ID for chlor_a
Read Error: 0 (/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.ncû^O) 0attachment 1
I looked for invisibles but could not find anything. I opened the input lists with various text editors, and looked for hidden symbols, but nothing.
I downloaded and processed again the same OLCI L1 to L2 but only keeping the two chlorophyll to see if it was aot_865 but I got the same error but for chlor_a with the same exit code:
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 523: NetCDF: Not a valid ID for chlor_a
Read Error: 0 (/home/gbourdin/test/S3A_OL_1_EFR____20161102T211755_20161102T212055_20180315T195348_0179_010_271_3240_LR2_R_NT_002.L2_SEN3.ncû^O) 0attachment 1
l2bin doesn't work with OLCI and MSI
Looking at the python code, you are passing the .txt file as the argument to "ifile=", so the error may be the netcdf library trying to open the .txt file. You might find it better to create a parameter file for l2gen.