how to use code to run l2gen correctly

[sawyer223]

hello
I am a newcomer to learning water color remote sensing. Recently, I am using the L2gen module in Seadas software for data processing, I am trying to use the L2gen module with Python for processing, but according to the l2gen help page, the code I wrote cannot get useful L2 data, and the data I get does not contain useful remote sensing reflectance band information, the following is my code, I hope to get your help. Thank you so much.
btw, my seadas version is 8.4.0 on Mac


import subprocess
import os

os.environ['OCDATAROOT'] = '/Users/sawyerduan/seadas/ocssw'
os.environ['OCVARROOT'] = '/Users/sawyerduan/seadas/ocssw'

l1b_file = "/Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf"
geo_file = "/Users/sawyerduan/Downloads/MYD03.A2023289.0550.061.2023289210348.hdf"

output_dir = "/Users/sawyerduan/Downloads/out"
l1b_timestamp = "20231030"

l2gen_command =f"/Users/sawyerduan/seadas/ocssw/bin/l2gen ifile={l1b_file} l2prod=Rrs_555,Rrs_645,Rrs_vvv,nLw_1240,nLw_1640,nLw_2130,nLw_748,nLw_859,nLw_869,nLw_vvv,a_vvv_carder,adg_vvv_carder,aph_vvv_carder,bbp_vvv_carder,cloud_albedo geofile={geo_file} ofile={output_dir}/{l1b_timestamp}.L2.OC.nc ",

try:
subprocess.run(l2gen_command,shell=True)
print("L2 processing completed successfully.")
except subprocess.CalledProcessError as e:
print(f"Error executing l2gen: {e.stderr}")

[OB SeaDAS - xuanyang02]

Can you do

Code: Select all

seadas_info

and share the result here?

Is your Mac with M1/M2 chip? If so, you need take extra steps to set up the environment in order to run l2gen https://seadas.gsfc.nasa.gov/downloads/

Or you can try to run l2gen using Client Server model (docker is recommended)

[sawyer223]

sorry to bother you .ok,my laptap is the MBA with M2 chip,and i have installed the ocssw successfully by following the install-step on the official website.When I use the l2gen in the seadas,it could get the right data with the Rrs and could calculate the SSC. But i want to use the code by pychrm or terminal to run the l2gen ,which can save time,and i have write the code with the l2gen-help pages.such as the ifile,ofile,geofile,even i got the L2 data. but when i use the data to calculate the SSC,and i find that there is not containing the data of Rrs,that is very strange.
i guess maybe i did not input the right parameters, but i did not find some template in the l2gen-help pages,like l2gen=ifile={l1b_file} l2prod=Rrs_555 geofile={geo_file} ofile={output_dir}/{l1b_timestamp}.L2.OC.nc So i did not know where should i put the parameters and how to list the right parameters.
the results of my l2 data. it is just 300+ kb, i think it did not contain the right things
if you could help me check up the code and find the error, i would be very appreciate.thank you very much

[OB SeaDAS - xuanyang02]

Code: Select all

l2gen ifile=/Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf geofile=/Users/sawyerduan/Downloads/MYD03.A2023289.0550.061.2023289210348.hdf l2prod="Rrs_vvv a_vvv_carder adg_vvv_carder aph_vvv_carder bbp_vvv_carder cloud_albedo nLw_1240 nLw_1640 nLw_2130 nLw_748 nLw_859 nLw_869 nLw_vvv" ofile=/Users/sawyerduan/Downloads/AQUA_MODIS.20231016T055000.L2.OC.nc

[sawyer223]

I am very glad that you gave me the example, and very thank you.
I run it in the terminal, and I got the l2 data,but it still does not contain anything, the size of that data is only 150kb,and i even couldn't open it in the seadas APP. it is really weird, i am very sorry that i still didn't make it and sorry to bother you. but i really want to figure it out, i hope you could read the following things that i run in the terminal.thank you!

(base) sawyerduan@SawyerdeMacBook-Air ~ % /Users/sawyerduan/seadas/ocssw/bin/l2gen ifile=/Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf geofile=/Users/sawyerduan/Downloads/MYD03.A2023289.0550.061.2023289210348.hdf l2prod="Rrs_645 Rrs_555 Rrs_2130 Rrs_748 Rrs_859 Rrs_869 Rrs_vvv " ofile=/Users/sawyerduan/Downloads/AQUA_MODIS.20231016T055000.L2.OC.nc maskglint=0 maskhilt=1 maskcloud=1 cloud_thresh=0.027 cloud_wave=1240 aer_opt=-3 aer_swir_long=2130 aer_swir_short=1240 aer_wave_long=869 aer_wave_short=748

Loading default parameters from /Users/sawyerduan/seadas/ocssw/common/msl12_defaults.par
Input file /Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf is MODIS Aqua Level-1B HDF-EOS product.

Loading characteristics for MODISA
Opening sensor information file /Users/sawyerduan/seadas/ocssw/modis/aqua/msl12_sensor_info.dat
Bnd Lam Fo Tau_r k_oz k_no2 t_co2 awhite aw bbw
0 412.000 172.912 3.099e-01 1.987e-03 5.814e-19 1.000e+00 1.000e+00 2.030e-02 2.884e-03
1 443.000 187.622 2.367e-01 3.189e-03 4.985e-19 1.000e+00 1.000e+00 1.110e-02 2.181e-03
2 469.000 205.878 1.914e-01 8.745e-03 3.938e-19 1.000e+00 1.000e+00 1.034e-02 1.749e-03
3 488.000 194.933 1.592e-01 2.032e-02 2.878e-19 1.000e+00 1.000e+00 1.676e-02 1.447e-03
4 531.000 185.747 1.126e-01 6.838e-02 1.525e-19 1.000e+00 1.000e+00 4.494e-02 1.014e-03
5 547.000 186.539 9.906e-02 8.622e-02 1.194e-19 1.000e+00 1.000e+00 5.491e-02 8.886e-04
6 555.000 183.869 9.432e-02 9.553e-02 9.445e-20 1.000e+00 1.000e+00 5.839e-02 8.443e-04
7 645.000 157.811 5.082e-02 7.382e-02 1.382e-20 1.000e+00 9.133e-01 3.458e-01 4.469e-04
8 667.000 152.255 4.443e-02 4.890e-02 7.065e-21 1.000e+00 8.921e-01 4.923e-01 3.900e-04
9 678.000 148.052 4.146e-02 3.787e-02 8.304e-21 1.000e+00 8.783e-01 5.116e-01 3.634e-04
10 748.000 128.065 2.849e-02 1.235e-02 2.157e-21 1.000e+00 7.832e-01 2.800e+00 2.470e-04
11 859.000 97.174 1.613e-02 2.347e-03 6.212e-23 1.000e+00 6.519e-01 4.396e+00 1.384e-04
12 869.000 95.824 1.540e-02 1.936e-03 7.872e-23 1.000e+00 6.403e-01 4.727e+00 1.320e-04
13 1240.000 45.467 3.617e-03 0.000e+00 0.000e+00 9.994e-01 0.000e+00 1.147e+02 3.021e-05
14 1640.000 23.977 1.219e-03 0.000e+00 0.000e+00 9.890e-01 0.000e+00 6.689e+02 1.000e-05
15 2130.000 9.885 4.286e-04 0.000e+00 0.000e+00 9.696e-01 0.000e+00 2.810e+03 3.465e-06

Bnd Lam
16 3750.000
17 3959.000
18 4050.000
19 6715.000
20 7325.000
21 8550.000
22 11000.000
23 12000.000

Loading default parameters for MODISA from /Users/sawyerduan/seadas/ocssw/modis/msl12_defaults.par
Loading default sub-sensor parameters for MODISA from /Users/sawyerduan/seadas/ocssw/modis/aqua/msl12_defaults.par
Loading parameters for suite OC from /Users/sawyerduan/seadas/ocssw/modis/msl12_defaults_OC.par
Loading command line parameters

Loading user parameters for MODISA

Internal data compression requested at compression level: 4
Opening filter file /Users/sawyerduan/seadas/ocssw/modis/msl12_filter.dat
Setting 3 x 3 straylight filter on HILT mask

Filter Kernel
1 1 1
1 1 1
1 1 1

Minimum fill set to 1 pixels


Setting 3 x 3 straylight filter on CLDICE mask

Filter Kernel
1 1 1
1 1 1
1 1 1

Minimum fill set to 1 pixels


Reading Thuillier_F0.dat
Processing at 1000 meter resolution.
1000-meter file: /Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf

Allocated 2224686 bytes in L1 record.
Allocated 844896 bytes in L2 record.

Opening: /Users/sawyerduan/Downloads/AQUA_MODIS.20231016T055000.L2.OC.nc


The following products will be included in /Users/sawyerduan/Downloads/AQUA_MODIS.20231016T055000.L2.OC.nc.
0 Rrs_645
1 Rrs_555
2 Rrs_2130
3 Rrs_748
4 Rrs_859
5 Rrs_869
6 Rrs_412
7 Rrs_443
8 Rrs_469
9 Rrs_488
10 Rrs_531
11 Rrs_547
12 Rrs_667
13 Rrs_678
14 l2_flags



Begin l2gen Version 9.6.0-V2023.3 Processing
Sensor is MODISA
Sensor ID is 7
Sensor has 16 reflective bands
Sensor has 8 emissive bands
Number of along-track detectors per band is 10
Number of input pixels per scan is 1354
Processing pixels 1 to 1354 by 1
Processing scans 1 to 2030 by 1
Ocean processing enabled
Land processing enabled
Atmospheric correction enabled

Begin MSl12 processing at 2023305220141000

Allocated 2224686 bytes in L1 record.
Allocated 2224686 bytes in L1 record.
Allocated 2224686 bytes in L1 record.
-E- /Users/dshea/ocssw/oel_util/libtimeutils/leapsecond.c:32 - leap second file (/Users/sawyerduan/seadas/ocssw/common/tai-utc.dat) not found.
(base) sawyerduan@SawyerdeMacBook-Air ~ %

[OB SeaDAS - xuanyang02]

The file not found is: /Users/sawyerduan/seadas/ocssw/var/common/tai-utc.dat

Try

Code: Select all

update_luts modisa

Hope this post helps -- viewtopic.php?t=3184

[sawyer223]

yes, it helps,thank you very much. i have been puzzled that error for sevaral days, but i just thought it is gonna to adjust the time lag, and not a big deal.
but it seems to occur another bug, and i still could not search its solution on the internet, i hope to get your guide. thanks!
btw, i have made the xcal.c in the right folder, the file path is right too.

Loaded monthly NSIDC ice climatology HDF file.
Loading XCAL rvs and polarization sensitivities from _412.hdf
-E- /Users/dshea/ocssw/oel_hdf4/libl1/xcal.c line 104: SDstart(_412.hdf, 1) failed.

[sawyer223]

hello
here is another part that i want to konw, i have multiple MODIS L1 files to generate L2 products. and i saw the posts in viewtopic.php?t=3198
then i tried multilevel_processor, i make the list by macvim,like this
/Users/sawyerduan/Downloads/l1b/MYD021KM.A2023278.0445.061.2023278165309.hdf
/Users/sawyerduan/Downloads/l1b/MYD021KM.A2023287.0605.061.2023287191448.hdf
/Users/sawyerduan/Downloads/l1b/MYD021KM.A2023289.0550.061.2023290134619.hdf
and i chose the

use_ancillary=1

and i make the content like this

[main]
ifile=/Users/sawyerduan/Downloads/vim
use_ancillary=1

[l2gen]
aer_opt=-3

and i got the error

execution exception: java.io.IOException: multilevel_processor failed with exit code 1.
Check log for more details.
Traceback (most recent call last):
File "/Users/sawyerduan/seadas/ocssw/bin/multilevel_processor", line 52, in __init__
os.mkdir(hidden_dir)
OSError: [Errno 30] Read-only file system: '.seadas_data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/sawyerduan/seadas/ocssw/bin/multilevel_processor", line 2515, in <module>
sys.exit(main())
^^^^^^
File "/Users/sawyerduan/seadas/ocssw/bin/multilevel_processor", line 2204, in main
cfg_data = ProcessorConfig('.seadas_data', os.getcwd(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sawyerduan/seadas/ocssw/bin/multilevel_processor", line 54, in __init__
if sys.exc_info()[1].find('Permission denied:') != -1:
^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'OSError' object has no attribute 'find'

i want to know what the .seadas.data's mean is ? the .L1A_LAC?

thanks

[OB SeaDAS - xuanyang02]

L2 needs an L1B file + GEO file as input

If /Users/sawyerduan/Downloads/vim is a list of L1B files, they have to follow the OBPG File Naming Convention https://oceancolor.gsfc.nasa.gov/resources/docs/filenaming-convention/,
such as AQUA_MODIS.20231014T060500.L1B.hdf,
with the corresponding GEO file AQUA_MODIS.20231014T060500.GEO.hdf in the same directory. Otherwise, multileve_processor will fail to find the corresponding GEO files, and then fail during the l2gen process. MYD021KM.A2023278.0445.061.2023278165309.hdf does not have "L1B" in the name, and mlp is not smart enough to know MYD03.A2023289.0550.061.2023289210348.hdf is the corresponding GEO file.

If /Users/sawyerduan/Downloads/vim is a list L1A files, such as A2023287060500.L1A_LAC at the OB.DAAC site -- https://oceandata.sci.gsfc.nasa.gov/directdataaccess/Level-1A/Aqua-MODIS/2003/278/
multilevel_processor will create GEO, L1B and L2 files.

[sawyer223]

hi,wish you a happy day!
i used the multilevel_processor, and i made a list of L1a files, unfortunately, i still didn't run l2gen smoothly. Although i used the plaintext, the error:ERROR!! Ifile does not exist showed again. I will post the picture of the list file's type to show you. i made the list by what you said,but it still couldnot run properly, is there something i missed? hope for your help, thank you again and again.