gpt MosaicGeneral

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

gpt MosaicGeneral

by avmehta » Thu Feb 11, 2021 2:46 pm America/New_York

Hello,
  I am doing mosaic of a few mages and want to set certain value to NaN
  I followed this statement in .par file:
  band1Expression=if (Kd_490 != 0) then 1.3/Kd_490 else NaN
   and have a statement that looks like:
   band1Expression=if (rhos_443 != 0) then rhos_443 else NaN
But I still get 0s and not NaN.
Not sure what I am doing wrong.
Thank you.
Best,
Amita

Tags:

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

gpt MosaicGeneral

by gnwiii » Thu Feb 11, 2021 6:55 pm America/New_York

You need to provide enough information so we can easily reproduce the problem.   It would be useful to know the file format you are using and whether NaN's are in locations that were not valid data in the input file (a report in the SNAP forum of scattered NaN's using gpt graph processing and writing to NetCDF4-BEAM that suggests "valid" pixels are being converted to NaN's).  

In the standard SNPP_VIIRS level-2 OC NetCDF files, Kd_490 not-valid values display as NaN's in the SeaDAS GUI using the  "Pixel Info" panel, but the band Properties show:
Valid-Pixel Expression  Kd_490 >= 0.01000 && Kd_490 <= 6.00000
No-Data Value Used  true
No-Data Value  -32767.0

In the NetCDF metadate these translate to:
        short Kd_490(number_of_lines, pixels_per_line) ;
                Kd_490:long_name = "Diffuse attenuation coefficient at 490 nm, KD2 algorithm" ;
                Kd_490:scale_factor = 0.0002f ;
                Kd_490:add_offset = 0.f ;
                Kd_490:units = "m^-1" ;
                Kd_490:_FillValue = -32767s ;
                Kd_490:valid_min = 50s ;
                Kd_490:valid_max = 30000s ;

The valid pixel expression for Kd_490 is based on ocssw/share/common/product.xml, but the entry in the same file for rhos is missing the validMin and validMax values.   NetCDF readers often translate the _FillValue to NaN when loading data and applying the offset and scale.

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

gpt MosaicGeneral

by avmehta » Fri Feb 12, 2021 9:44 am America/New_York

Hello George,
  Thank you for the explanation. What I want to do is exclude land and cloud pixels from mosaic/composite and then set rhos to NaN or some missing value number if the value is 0. My par file looks like this:

band1=rhos_412
band1Expression=if (rhos_412 != 0) then rhos_412 else NaN
qualityExpression=!l2_flags.LAND and !l2_flags.CLDICE
north=39.8
south=36.5
west=-77.01
east=-75.66
pixelSize=0.009
resampling=Nearest
crs=GEOGCS["WGS84(DD)", DATUM["WGS84", SPHEROID["WGS84", 6378137.0, 298.257223563]], PRIMEM["Greenwich", 0.0], UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST], AXIS["Geodetic latitude", NORTH]]

And the xml file :
<graph id="MosaicGraph">
  <version>1.0</version>

  <node id="MosaicNode">
    <operator>Mosaic</operator>
    <sources>
       <sourceProducts>${sourceProducts}</sourceProducts>
    </sources>

    <parameters>
        <variables>
            <variable>
                <name>${band1}</name>
                <expression>${band1Expression}</expression>
            </variable>

        </variables>
        <conditions>
            <condition>
                <name>condition_0</name>
                <expression>${qualityExpression}</expression>
                <output>false</output>
            </condition>
        </conditions>
        <combine>OR</combine>
        <crs>${crs}</crs>
        <resampling>${resampling}</resampling>
        <westBound>${west}</westBound>
        <northBound>${north}</northBound>
        <eastBound>${east}</eastBound>
        <southBound>${south}</southBound>
        <pixelSizeX>${pixelSize}</pixelSizeX>
        <pixelSizeY>${pixelSize}</pixelSizeY>
    </parameters>

Thank you
Amita

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

gpt MosaicGeneral

by gnwiii » Fri Feb 12, 2021 10:43 am America/New_York

We still don't know what file formats you are using (metadata handling is not consistent across formats) and how the NaN values in the output images relate to missing or flagged pixels in the level-2 file.   Can you select a small area in the output image with NaN and valid pixels?  If so you can inspect the same region in the input level-2 files to get an understanding of what triggers the NaN values.   It might be helpful to modify the product.xml file to add the valid min and max for entries for rhos so these appear in the GUI properties window.

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

gpt MosaicGeneral

by avmehta » Fri Feb 12, 2021 12:30 pm America/New_York

In the GUI I can see NaN where there is no data (Rhos =0  there show up as NaN) it is the NetCDF file that they stay 0s and no appear as NaN. I tried a number (-9999) instead of NaN but that does not work either!
Thanks much for your help.
Amita

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

gpt MosaicGeneral

by gnwiii » Fri Feb 12, 2021 5:18 pm America/New_York

Where are you seeing rhos=0?   In the calc_rhos.c source I see l1rec->rhos[ipb] = BAD_FLT; unless l1rec->Lt[ipb] <= 0., where l12_parms.h contains #define BAD_FLT  -32767.0.

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

gpt MosaicGeneral

by avmehta » Sat Feb 13, 2021 1:23 pm America/New_York

I just look at ncdump of the L2 nc file.
Thanks.
Amita

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

gpt MosaicGeneral

by gnwiii » Sat Feb 13, 2021 5:08 pm America/New_York

I used the SeaDAS benchmark files (A2006167181000.L1B and A2006167181000.GEO) to get rhos_443.  ncdump -h shows:
        float rhos_443(number_of_lines, pixels_per_line) ;
                rhos_443:long_name = "Surface reflectance at 443 nm" ;
                rhos_443:standard_name = "surface_albedo" ;
                rhos_443:_FillValue = -32767.f ;
                rhos_443:valid_min = -0.05f ;
                rhos_443:valid_max = 1.f ;

so zero is considered "valid".

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

gpt MosaicGeneral

by avmehta » Mon Feb 15, 2021 10:04 am America/New_York

Thanks. Here is a portion of what I get after I use gpt to mosaic files:

metadata:Processing_Graph:node_0:parameters:orthorectify = "false" ;
                metadata:Processing_Graph:node_0:parameters:combine = "OR" ;
                metadata:Processing_Graph:node_0:parameters:variables:variable:name = "rhos_412" ;
                metadata:Processing_Graph:node_0:parameters:variables:variable:expression = "if (rhos_412 != 0) then rhos_412 else -999" ;
                metadata:Processing_Graph:node_0:parameters:conditions:condition:name = "condition_0" ;
                metadata:Processing_Graph:node_0:parameters:conditions:condition:output = "false" ;
                metadata:Processing_Graph:node_0:parameters:conditions:condition:expression = "!l2_flags.LAND and !l2_flags.CLDICE" ;
        float rhos_412(lat, lon) ;
                rhos_412:long_name = "if (rhos_412 != 0) then rhos_412 else -999" ;
                rhos_412:coordinates = "lat lon" ;
                rhos_412:valid_pixel_expression = "rhos_412_count > 0" ;

All I wan to do is set '0' values to some number or NaN.
Thanks.
Amita

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

gpt MosaicGeneral

by gnwiii » Mon Feb 15, 2021 11:07 am America/New_York

Try looking at the histogram in the SeaDAS GUI.  My test image has only 6 values below 0.015, and no zeros.  In the GUI, the FillValue (-32767.0) is translated to NaN, but if you are using a different file format or reader the FillValue may be left as -32767.0.

Post Reply