how to use code to run l2gen correctly

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
sawyer223
Posts: 5
Joined: Mon Oct 30, 2023 4:43 am America/New_York
Answers: 0

how to use code to run l2gen correctly

by sawyer223 » Mon Oct 30, 2023 4:56 am America/New_York

hello
I am a newcomer to learning water color remote sensing. Recently, I am using the L2gen module in Seadas software for data processing, I am trying to use the L2gen module with Python for processing, but according to the l2gen help page, the code I wrote cannot get useful L2 data, and the data I get does not contain useful remote sensing reflectance band information, the following is my code, I hope to get your help. Thank you so much.
btw, my seadas version is 8.4.0 on Mac


import subprocess
import os

os.environ['OCDATAROOT'] = '/Users/sawyerduan/seadas/ocssw'
os.environ['OCVARROOT'] = '/Users/sawyerduan/seadas/ocssw'

l1b_file = "/Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf"
geo_file = "/Users/sawyerduan/Downloads/MYD03.A2023289.0550.061.2023289210348.hdf"

output_dir = "/Users/sawyerduan/Downloads/out"
l1b_timestamp = "20231030"

l2gen_command =f"/Users/sawyerduan/seadas/ocssw/bin/l2gen ifile={l1b_file} l2prod=Rrs_555,Rrs_645,Rrs_vvv,nLw_1240,nLw_1640,nLw_2130,nLw_748,nLw_859,nLw_869,nLw_vvv,a_vvv_carder,adg_vvv_carder,aph_vvv_carder,bbp_vvv_carder,cloud_albedo geofile={geo_file} ofile={output_dir}/{l1b_timestamp}.L2.OC.nc ",

try:
subprocess.run(l2gen_command,shell=True)
print("L2 processing completed successfully.")
except subprocess.CalledProcessError as e:
print(f"Error executing l2gen: {e.stderr}")

Filters:

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 660
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: how to use code to run l2gen correctly

by OB SeaDAS - xuanyang02 » Mon Oct 30, 2023 1:47 pm America/New_York

Can you do

Code: Select all

seadas_info
and share the result here?

Is your Mac with M1/M2 chip? If so, you need take extra steps to set up the environment in order to run l2gen https://seadas.gsfc.nasa.gov/downloads/

Or you can try to run l2gen using Client Server model (docker is recommended)

sawyer223
Posts: 5
Joined: Mon Oct 30, 2023 4:43 am America/New_York
Answers: 0

Re: how to use code to run l2gen correctly

by sawyer223 » Mon Oct 30, 2023 9:22 pm America/New_York

sorry to bother you .ok,my laptap is the MBA with M2 chip,and i have installed the ocssw successfully by following the install-step on the official website.When I use the l2gen in the seadas,it could get the right data with the Rrs and could calculate the SSC. But i want to use the code by pychrm or terminal to run the l2gen ,which can save time,and i have write the code with the l2gen-help pages.such as the ifile,ofile,geofile,even i got the L2 data. but when i use the data to calculate the SSC,and i find that there is not containing the data of Rrs,that is very strange.
i guess maybe i did not input the right parameters, but i did not find some template in the l2gen-help pages,like l2gen=ifile={l1b_file} l2prod=Rrs_555 geofile={geo_file} ofile={output_dir}/{l1b_timestamp}.L2.OC.nc So i did not know where should i put the parameters and how to list the right parameters.
the results of my l2 data. it is just 300+ kb, i think it did not contain the right things
if you could help me check up the code and find the error, i would be very appreciate.thank you very much
Attachments
the l2 file's preview
the l2 file's preview
截屏2023-10-31 09.16.17.png (65.98 KiB) Not viewed yet

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 660
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: how to use code to run l2gen correctly

by OB SeaDAS - xuanyang02 » Tue Oct 31, 2023 11:09 pm America/New_York

Code: Select all

l2gen ifile=/Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf geofile=/Users/sawyerduan/Downloads/MYD03.A2023289.0550.061.2023289210348.hdf l2prod="Rrs_vvv a_vvv_carder adg_vvv_carder aph_vvv_carder bbp_vvv_carder cloud_albedo nLw_1240 nLw_1640 nLw_2130 nLw_748 nLw_859 nLw_869 nLw_vvv" ofile=/Users/sawyerduan/Downloads/AQUA_MODIS.20231016T055000.L2.OC.nc

sawyer223
Posts: 5
Joined: Mon Oct 30, 2023 4:43 am America/New_York
Answers: 0

Re: how to use code to run l2gen correctly

by sawyer223 » Wed Nov 01, 2023 10:07 am America/New_York

I am very glad that you gave me the example, and very thank you.
I run it in the terminal, and I got the l2 data,but it still does not contain anything, the size of that data is only 150kb,and i even couldn't open it in the seadas APP. it is really weird, i am very sorry that i still didn't make it and sorry to bother you. but i really want to figure it out, i hope you could read the following things that i run in the terminal.thank you!

(base) sawyerduan@SawyerdeMacBook-Air ~ % /Users/sawyerduan/seadas/ocssw/bin/l2gen ifile=/Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf geofile=/Users/sawyerduan/Downloads/MYD03.A2023289.0550.061.2023289210348.hdf l2prod="Rrs_645 Rrs_555 Rrs_2130 Rrs_748 Rrs_859 Rrs_869 Rrs_vvv " ofile=/Users/sawyerduan/Downloads/AQUA_MODIS.20231016T055000.L2.OC.nc maskglint=0 maskhilt=1 maskcloud=1 cloud_thresh=0.027 cloud_wave=1240 aer_opt=-3 aer_swir_long=2130 aer_swir_short=1240 aer_wave_long=869 aer_wave_short=748

Loading default parameters from /Users/sawyerduan/seadas/ocssw/common/msl12_defaults.par
Input file /Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf is MODIS Aqua Level-1B HDF-EOS product.

Loading characteristics for MODISA
Opening sensor information file /Users/sawyerduan/seadas/ocssw/modis/aqua/msl12_sensor_info.dat
Bnd Lam Fo Tau_r k_oz k_no2 t_co2 awhite aw bbw
0 412.000 172.912 3.099e-01 1.987e-03 5.814e-19 1.000e+00 1.000e+00 2.030e-02 2.884e-03
1 443.000 187.622 2.367e-01 3.189e-03 4.985e-19 1.000e+00 1.000e+00 1.110e-02 2.181e-03
2 469.000 205.878 1.914e-01 8.745e-03 3.938e-19 1.000e+00 1.000e+00 1.034e-02 1.749e-03
3 488.000 194.933 1.592e-01 2.032e-02 2.878e-19 1.000e+00 1.000e+00 1.676e-02 1.447e-03
4 531.000 185.747 1.126e-01 6.838e-02 1.525e-19 1.000e+00 1.000e+00 4.494e-02 1.014e-03
5 547.000 186.539 9.906e-02 8.622e-02 1.194e-19 1.000e+00 1.000e+00 5.491e-02 8.886e-04
6 555.000 183.869 9.432e-02 9.553e-02 9.445e-20 1.000e+00 1.000e+00 5.839e-02 8.443e-04
7 645.000 157.811 5.082e-02 7.382e-02 1.382e-20 1.000e+00 9.133e-01 3.458e-01 4.469e-04
8 667.000 152.255 4.443e-02 4.890e-02 7.065e-21 1.000e+00 8.921e-01 4.923e-01 3.900e-04
9 678.000 148.052 4.146e-02 3.787e-02 8.304e-21 1.000e+00 8.783e-01 5.116e-01 3.634e-04
10 748.000 128.065 2.849e-02 1.235e-02 2.157e-21 1.000e+00 7.832e-01 2.800e+00 2.470e-04
11 859.000 97.174 1.613e-02 2.347e-03 6.212e-23 1.000e+00 6.519e-01 4.396e+00 1.384e-04
12 869.000 95.824 1.540e-02 1.936e-03 7.872e-23 1.000e+00 6.403e-01 4.727e+00 1.320e-04
13 1240.000 45.467 3.617e-03 0.000e+00 0.000e+00 9.994e-01 0.000e+00 1.147e+02 3.021e-05
14 1640.000 23.977 1.219e-03 0.000e+00 0.000e+00 9.890e-01 0.000e+00 6.689e+02 1.000e-05
15 2130.000 9.885 4.286e-04 0.000e+00 0.000e+00 9.696e-01 0.000e+00 2.810e+03 3.465e-06

Bnd Lam
16 3750.000
17 3959.000
18 4050.000
19 6715.000
20 7325.000
21 8550.000
22 11000.000
23 12000.000

Loading default parameters for MODISA from /Users/sawyerduan/seadas/ocssw/modis/msl12_defaults.par
Loading default sub-sensor parameters for MODISA from /Users/sawyerduan/seadas/ocssw/modis/aqua/msl12_defaults.par
Loading parameters for suite OC from /Users/sawyerduan/seadas/ocssw/modis/msl12_defaults_OC.par
Loading command line parameters

Loading user parameters for MODISA

Internal data compression requested at compression level: 4
Opening filter file /Users/sawyerduan/seadas/ocssw/modis/msl12_filter.dat
Setting 3 x 3 straylight filter on HILT mask

Filter Kernel
1 1 1
1 1 1
1 1 1

Minimum fill set to 1 pixels


Setting 3 x 3 straylight filter on CLDICE mask

Filter Kernel
1 1 1
1 1 1
1 1 1

Minimum fill set to 1 pixels


Reading Thuillier_F0.dat
Processing at 1000 meter resolution.
1000-meter file: /Users/sawyerduan/Downloads/MYD021KM.A2023289.0550.061.2023290134619.hdf

Allocated 2224686 bytes in L1 record.
Allocated 844896 bytes in L2 record.

Opening: /Users/sawyerduan/Downloads/AQUA_MODIS.20231016T055000.L2.OC.nc


The following products will be included in /Users/sawyerduan/Downloads/AQUA_MODIS.20231016T055000.L2.OC.nc.
0 Rrs_645
1 Rrs_555
2 Rrs_2130
3 Rrs_748
4 Rrs_859
5 Rrs_869
6 Rrs_412
7 Rrs_443
8 Rrs_469
9 Rrs_488
10 Rrs_531
11 Rrs_547
12 Rrs_667
13 Rrs_678
14 l2_flags



Begin l2gen Version 9.6.0-V2023.3 Processing
Sensor is MODISA
Sensor ID is 7
Sensor has 16 reflective bands
Sensor has 8 emissive bands
Number of along-track detectors per band is 10
Number of input pixels per scan is 1354
Processing pixels 1 to 1354 by 1
Processing scans 1 to 2030 by 1
Ocean processing enabled
Land processing enabled
Atmospheric correction enabled

Begin MSl12 processing at 2023305220141000

Allocated 2224686 bytes in L1 record.
Allocated 2224686 bytes in L1 record.
Allocated 2224686 bytes in L1 record.
-E- /Users/dshea/ocssw/oel_util/libtimeutils/leapsecond.c:32 - leap second file (/Users/sawyerduan/seadas/ocssw/common/tai-utc.dat) not found.
(base) sawyerduan@SawyerdeMacBook-Air ~ %
Attachments
截屏2023-11-01 22.03.10.png
截屏2023-11-01 22.03.10.png (241.92 KiB) Not viewed yet

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 660
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: how to use code to run l2gen correctly

by OB SeaDAS - xuanyang02 » Wed Nov 01, 2023 4:28 pm America/New_York

The file not found is: /Users/sawyerduan/seadas/ocssw/var/common/tai-utc.dat

Try

Code: Select all

update_luts modisa
Hope this post helps -- viewtopic.php?t=3184

sawyer223
Posts: 5
Joined: Mon Oct 30, 2023 4:43 am America/New_York
Answers: 0

Re: how to use code to run l2gen correctly

by sawyer223 » Wed Nov 01, 2023 9:18 pm America/New_York

yes, it helps,thank you very much. i have been puzzled that error for sevaral days, but i just thought it is gonna to adjust the time lag, and not a big deal.
but it seems to occur another bug, and i still could not search its solution on the internet, i hope to get your guide. thanks!
btw, i have made the xcal.c in the right folder, the file path is right too.

Loaded monthly NSIDC ice climatology HDF file.
Loading XCAL rvs and polarization sensitivities from _412.hdf
-E- /Users/dshea/ocssw/oel_hdf4/libl1/xcal.c line 104: SDstart(_412.hdf, 1) failed.

sawyer223
Posts: 5
Joined: Mon Oct 30, 2023 4:43 am America/New_York
Answers: 0

Re: how to use code to run l2gen correctly

by sawyer223 » Wed Nov 01, 2023 10:55 pm America/New_York

hello
here is another part that i want to konw, i have multiple MODIS L1 files to generate L2 products. and i saw the posts in viewtopic.php?t=3198
then i tried multilevel_processor, i make the list by macvim,like this
/Users/sawyerduan/Downloads/l1b/MYD021KM.A2023278.0445.061.2023278165309.hdf
/Users/sawyerduan/Downloads/l1b/MYD021KM.A2023287.0605.061.2023287191448.hdf
/Users/sawyerduan/Downloads/l1b/MYD021KM.A2023289.0550.061.2023290134619.hdf
and i chose the

use_ancillary=1

and i make the content like this

[main]
ifile=/Users/sawyerduan/Downloads/vim
use_ancillary=1

[l2gen]
aer_opt=-3

and i got the error

execution exception: java.io.IOException: multilevel_processor failed with exit code 1.
Check log for more details.
Traceback (most recent call last):
File "/Users/sawyerduan/seadas/ocssw/bin/multilevel_processor", line 52, in __init__
os.mkdir(hidden_dir)
OSError: [Errno 30] Read-only file system: '.seadas_data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/sawyerduan/seadas/ocssw/bin/multilevel_processor", line 2515, in <module>
sys.exit(main())
^^^^^^
File "/Users/sawyerduan/seadas/ocssw/bin/multilevel_processor", line 2204, in main
cfg_data = ProcessorConfig('.seadas_data', os.getcwd(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sawyerduan/seadas/ocssw/bin/multilevel_processor", line 54, in __init__
if sys.exc_info()[1].find('Permission denied:') != -1:
^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'OSError' object has no attribute 'find'

i want to know what the .seadas.data's mean is ? the .L1A_LAC?

thanks

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 660
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: how to use code to run l2gen correctly

by OB SeaDAS - xuanyang02 » Thu Nov 02, 2023 10:24 am America/New_York

L2 needs an L1B file + GEO file as input

If /Users/sawyerduan/Downloads/vim is a list of L1B files, they have to follow the OBPG File Naming Convention https://oceancolor.gsfc.nasa.gov/resources/docs/filenaming-convention/,
such as AQUA_MODIS.20231014T060500.L1B.hdf,
with the corresponding GEO file AQUA_MODIS.20231014T060500.GEO.hdf in the same directory. Otherwise, multileve_processor will fail to find the corresponding GEO files, and then fail during the l2gen process. MYD021KM.A2023278.0445.061.2023278165309.hdf does not have "L1B" in the name, and mlp is not smart enough to know MYD03.A2023289.0550.061.2023289210348.hdf is the corresponding GEO file.

If /Users/sawyerduan/Downloads/vim is a list L1A files, such as A2023287060500.L1A_LAC at the OB.DAAC site -- https://oceandata.sci.gsfc.nasa.gov/directdataaccess/Level-1A/Aqua-MODIS/2003/278/
multilevel_processor will create GEO, L1B and L2 files.
Last edited by OB SeaDAS - xuanyang02 on Fri Nov 03, 2023 9:33 am America/New_York, edited 1 time in total.

sawyer223
Posts: 5
Joined: Mon Oct 30, 2023 4:43 am America/New_York
Answers: 0

Re: how to use code to run l2gen correctly

by sawyer223 » Thu Nov 02, 2023 10:42 am America/New_York

hi,wish you a happy day!
i used the multilevel_processor, and i made a list of L1a files, unfortunately, i still didn't run l2gen smoothly. Although i used the plaintext, the error:ERROR!! Ifile does not exist showed again. I will post the picture of the list file's type to show you. i made the list by what you said,but it still couldnot run properly, is there something i missed? hope for your help, thank you again and again.
Attachments
Screenshot 2023-11-03 at 09.45.42.png
Screenshot 2023-11-03 at 09.45.42.png (306.58 KiB) Not viewed yet
the files, the path and the list
the files, the path and the list
Screenshot 2023-11-03 at 09.24.43.png (233.07 KiB) Not viewed yet
Last edited by sawyer223 on Thu Nov 02, 2023 9:50 pm America/New_York, edited 2 times in total.

Post Reply