Seadas batch processing: Extracting l2flags and szen

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
madjidh
Posts: 40
Joined: Wed Feb 27, 2019 10:05 am America/New_York
Answers: 0
Has thanked: 1 time

Seadas batch processing: Extracting l2flags and szen

by madjidh » Fri Nov 26, 2021 10:23 am America/New_York

Hello,

I am batch processing some MODIS Aqua data (L1A to L3M, Rrs with the basic atmospheric correction) and I would like to include the szen and l2_flags.
I have 2 issues :
1) When I add szen, nothing happens and an error is returned saying szen does not exist. "-E- /home/seadas/ocssw/src/l2gen/l2_generic.c line 458: product szen not found." I can see different szen related to each waveband with the l2gen products window gui, but is there any general szen I can access to?
2) When I add l2_flags (note I cannot find it from the l2gen products window using the gui), I can see it and play with it using the L2 file created, but I cannot see it inside the L3M file at the end, only the Rrs and coordinates are created. See below an example of ncdump -h randomDay.L3m_DAY_OC with one Rrs processed:
variables:
short Rrs_859(lat, lon) ;
Rrs_859:long_name = "Remote sensing reflectance at 859 nm" ;
Rrs_859:scale_factor = 2.e-06f ;
Rrs_859:add_offset = 0.05f ;
Rrs_859:units = "sr^-1" ;
Rrs_859:standard_name = "surface_ratio_of_upwelling_radiance_emerging_from_sea_water_to_downwelling_radiative_flux_in_air" ;
Rrs_859:_FillValue = -32767s ;
Rrs_859:valid_min = -30000s ;
Rrs_859:valid_max = 25000s ;
Rrs_859:display_scale = "linear" ;
Rrs_859:display_min = 0.f ;
Rrs_859:display_max = 0.015f ;
float lat(lat) ;
lat:long_name = "Latitude" ;
lat:units = "degrees_north" ;
lat:standard_name = "latitude" ;
lat:_FillValue = -999.f ;
lat:valid_min = -90.f ;
lat:valid_max = 90.f ;
float lon(lon) ;
lon:long_name = "Longitude" ;
lon:units = "degrees_east" ;
lon:standard_name = "longitude" ;
lon:_FillValue = -999.f ;
lon:valid_min = -180.f ;
lon:valid_max = 180.f ;
ubyte palette(rgb, eightbitcolor) ;
I also did not ask for palette/eightbitcolor, but it is still returned, however I have no idea what it is.
Here are the lines I ran, it worked when I processed Rrs/Rhos/Rhot in the past but it does not work for szen/l2_flags:

Code: Select all

##########################################################################
################ START THE PROCESSING FROM L1A to L3m ####################
##########################################################################
echo "Start processing from L1A to L3m.."
cd $fullpath # Go to the folder where you need to process

################ START THE PROCESSING FROM L1A to GEO #####################
# Process the L1A subscene files to GEO
echo "modis_GEO : Processing L1A to GEO.."
modis_GEO.py $FILE -o $GEOFILE
echo "modis_GEO Done!"

################ START THE PROCESSING FROM L1A to L1B #####################
echo "modis_L1B : Processing Level 1A to Level 1B.."
# Process the L1A/GEO subscene files to L1B
modis_L1B.py $FILE $GEOFILE -o $L1BFILE --del-hkm --del-qkm
# Determine ancillary data
# Create a file in l2gen's par file format called L1BFILE.anc
getanc.py $L1BFILE
echo "modis_L1B Done!"
		
################ START PROCESSING FROM L1B to L2A #####################
echo "l2gen : Processing $L1BFILE to Level 2.."
l2gen ifile=$L1BFILE geofile=$GEOFILE ofile=$L2FILE \ l2prod1='Rrs_859,l2_flags' \
echo "l2gen Done!"
		
################ START PROCESSING FROM L2A to L3BIN #####################
echo "l2bin : Processing $L3BINFILE to Level 3 binned.."
l2bin ifile=$L2FILE ofile=$L3BINFILE \
resolve=1 \
flaguse=ATMFAIL \
echo "l2bin Done!"

################ START PROCESSING FROM L3BIN to L3MAP #####################
echo "l3mapgen : Processing $L3BINFILE to Level 3 mapped.."
l3mapgen ifile=$L3BINFILE ofile=$L3MAPFILE \
resolution=1km \
interp=area \
north=65 \
south=48 \
east=13 \
west=-25 \
fudge=3 \

Am I missing a detail ?
Thanks

Tags:

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

Re: Seadas batch processing: Extracting l2flags and szen

by gnwiii » Fri Nov 26, 2021 3:02 pm America/New_York

Today is the US Thanksgiving Holiday, but from Canada I can try to answer:

Mapping software is generally restricted to variables that can take on a continuous range of values. Flags and classification data have discrete values. Mapping requires assigning values of the variable by some form of interpolation or averaging, which can produce non-integer values.

I'm not sure why szen goes missing. Sometimes batch scripts and parameter files don't behave as expected due to simple typographic errors or "invisible" UniCode glyphs inserted by overly helpful editors.

madjidh
Posts: 40
Joined: Wed Feb 27, 2019 10:05 am America/New_York
Answers: 0
Has thanked: 1 time

Re: Seadas batch processing: Extracting l2flags and szen

by madjidh » Mon Nov 29, 2021 6:46 am America/New_York

Thanks for your answer George.

I was playing with the SeaBASS SST data set and we have access to l2flags with this data set, which is why I wanted to have access to it with my own matchup data set. If I am right, the satellite information inside SeaBASS comes from a 10x10 km grid around the in situ sample, which would mean that it has been reprojected, therefore I was hoping that a way to interpolate l2flags existed. I wouldn't be picky with it, if 1% of a reprojected pixel would be impacted by a mask that did not impact the other 99%, I would still include it.

For szen, I have no idea why it does not work yet.

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

Re: Seadas batch processing: Extracting l2flags and szen

by gnwiii » Mon Nov 29, 2021 8:26 am America/New_York

For comparisons with in situ data I would use unmapped level-2 pixels in some region around the in situ location for a pass within a time window around the time of the in situ measurement. Unwanted/dubious pixels can be excluded using flags, and you can cosider the distribution of satellite values in the region.

In the ocean you there are always gradients, so the range of values from a square patch will generally be greater than for a circular patch with the same area. Think about using 10x10 patches with corner pixels excluded. A 9x9 patch has 81 pixels, 10x10 with 4*6 corner pixels removed leaves 76, and will generally show reduced "spread".

The name "szen" is ambiguous (sun/sensor), so you need to find the product name. You can dump and ASCII xml format file with

Code: Select all

seadas@gwlaptop:~$ l2gen prodxmlfile=l2gen_prod.xml
Writing product information to XML file l2gen_prod.xml
There you can find level-2 product names for multiple zenith angle products.

madjidh
Posts: 40
Joined: Wed Feb 27, 2019 10:05 am America/New_York
Answers: 0
Has thanked: 1 time

Re: Seadas batch processing: Extracting l2flags and szen

by madjidh » Mon Nov 29, 2021 9:34 am America/New_York

The name "szen" is ambiguous (sun/sensor), so you need to find the product name. You can dump and ASCII xml format file with
Actually you could not be more right, I did a mistake, the real name is senz (sensor zenith angle), so my first problem is solved, thank you.

I already planned to use level 2 data of my matchups. I still wonder if there is a way to process l2flags at level 3 using seadas.

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Seadas batch processing: Extracting l2flags and szen

by OB.DAAC - SeanBailey » Mon Nov 29, 2021 10:23 am America/New_York

I still wonder if there is a way to process l2flags at level 3 using seadas
No. The l2_flags are used by l2bin when selecting data to include (or more to the point, exclude) in the binning process. The l2bin flaguse parameter defines which l2_flags to set as masks when binning. You cannot bin the l2_flags, as they are a bitwise value, not a float, so averaging it would produce a garbled, nonsensical result.

The SeaBASS SST matchup dataset is derived from L2 products, and are thus not binned (or reprojected), so the pixel-level l2_flags can be reported. The program "val_extract" (well, it's sst-specific variant "sstval_extract") is used in the SST matchup dataset generation.

Sean

madjidh
Posts: 40
Joined: Wed Feb 27, 2019 10:05 am America/New_York
Answers: 0
Has thanked: 1 time

Re: Seadas batch processing: Extracting l2flags and szen

by madjidh » Tue Nov 30, 2021 3:41 am America/New_York

Thank you for the details Sean !

Post Reply