l2bin/l3bin/l3mapgen distortion for non-standard data product

Please enter here to ask a question about any NASA Science related topics!
Post Reply
yonisherman
Posts: 11
Joined: Wed Mar 27, 2019 4:17 pm America/New_York

l2bin/l3bin/l3mapgen distortion for non-standard data product

by yonisherman » Wed Oct 27, 2021 10:31 am America/New_York

Hi all,
I am trying to test an MDN approach to retrieve chl a using MODIS Rrs (see Pahlevan et al. 2020, https://www.sciencedirect.com/science/a ... 5719306248).
I am using L2 data files I created using the standard l2gen (at 1000m resolution). Afterwards, I use the Rrs in the model and append the result in the L2 nc file as chl_mdn. In order to proceed with l2bin-l3bin-l3mapgen to get weekly composites I first added a parameter in the product.xml file, following a forum post I saw. When I plot the l2 data directly the data looks good and I can compare with the standard chl a product, however the chl_mdn mapped image is completely distorted (see attache image). I am not sure if I am using any of the steps incorrectly (standard parameters for all steps at 1km and 'regional' prodtype) , or perhaps my input in product.xml is incorrect (wasn't sure if need to be under the overarching chl product as an algorithm or as a separate product, currently I am using the latter, see below).

Do you have any suggestions about using non-standard products as far as binning goes.
Thank you so much
Jonathan Sherman


</product>
<product name="chl_mdn">
<standardName>mass_concentration_of_chlorophyll_in_sea_water dervied using MDN</standardName>
<units>mg m^-3</units>
<category>Derived</category>
<range>
<validMin>0.001</validMin>
<validMax>100</validMax>
<displayMin>0.01</displayMin>
<displayMax>20</displayMax>
</range>
<algorithm>
<cat_ix>-1</cat_ix><!-- The cat_ix is required by l2gen - Since you're not creating the product within l2gen, leave this alone -->
<description>Chl from MDN without 748nm</description>
<reference>Pahlevan, N., B. Smith, J. Schalles, et al. 2020. "Seamless retrievals of chlorophyll-a from Sentinel-2 (MSI) and Sentinel-3 (OLCI) in inland and coastal waters: A machine-learning approach." Remote Sensing of Environment, 111604 [10.1016/j.rse.2019.111604]</reference>
</algorithm>
</product>
Attachments
Screen Shot 2021-10-27 at 10.21.56 AM.pdf
l2 vs l3 comparison
(327.03 KiB) Downloaded 30 times
by gnwiii » Mon Nov 01, 2021 5:16 pm America/New_York
It still isn't clear to me whether the problem is that l2bin mishandled a 64-bit array or you stored a 32-bit array but labelled it as 64-bit data. The latter seems more likely as the dimensions of the array would be smaller by a factor of 2. Remote sensing often uses 32-bit arrays for geophysical data to save space while still providing more precision than most date warrant. Modern computers, however, generally do numerical calculations in 64-bit precision, so extra work may be needed to get a result as 32-bit data. I'll need to do some experiments with different languages -- the original calculation was done soon after netcdf4 support was added to the OCSSW software so libraries and languages have changed.
Go to full post

Tags:

gnwiii
Posts: 642
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

Re: l2bin/l3bin/l3mapgen distortion for non-standard data product

by gnwiii » Wed Oct 27, 2021 2:48 pm America/New_York

A number of years ago I was able to add a non-standard product to NASA level-2 files and did not encounter problems with mapping. My guess is that the xml file is not the cause. Looking at the NetCDF header with "ncdump -h" for differences between NASA chl variables and the new variable might be informative. You can ary using gdalinfo to compare the mapping between a NASA chl and your product.

yonisherman
Posts: 11
Joined: Wed Mar 27, 2019 4:17 pm America/New_York

Re: l2bin/l3bin/l3mapgen distortion for non-standard data product

by yonisherman » Wed Oct 27, 2021 10:40 pm America/New_York

Thank you for the reply. I am pretty sure I had a used the get_product_info command after editing the product.xml file to see if it was added, but I should double check to see how it fully compares with the standard chla product.

gnwiii
Posts: 642
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

Re: l2bin/l3bin/l3mapgen distortion for non-standard data product

by gnwiii » Thu Oct 28, 2021 9:10 am America/New_York

yonisherman wrote: Wed Oct 27, 2021 10:40 pm America/New_York Thank you for the reply. I am pretty sure I had a used the get_product_info command after editing the product.xml file to see if it was added, but I should double check to see how it fully compares with the standard chla product.
If the .xml is bad, the binning and mapping would complain, so I suspect the problem is with the coordinates. I guess you are using Panoply, so you might see a difference in the metadata for the level-2 file in Panoply's metadata display. You can also use l3bindump to inspect a few test locations. I expect you will find problems in the binned files.

yonisherman
Posts: 11
Joined: Wed Mar 27, 2019 4:17 pm America/New_York

Re: l2bin/l3bin/l3mapgen distortion for non-standard data product

by yonisherman » Fri Oct 29, 2021 12:34 pm America/New_York

Hi,
I just wanted to update on this if anyone in the future comes across a similar issue. As gnwiii commented, if l2bin/l3bin/l3mapgen are working, the problem isn't with the xml file where I added the new product/algorithm. Following gnwiii's suggestion, I used gdalinfo to inspect the netcdf l2 file passed into l2bin and onwards. It appeared that when I appended my custom chl product into the netcdf file I added it as a 64-bit floating-point rather then a 32-bit floating point used for the products produced by l2gen. After changing this the l3maps look normal.

Thanks gnwiii for the help.
Jonathan
Last edited by yonisherman on Fri Oct 29, 2021 12:35 pm America/New_York, edited 1 time in total.

gnwiii
Posts: 642
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

Re: l2bin/l3bin/l3mapgen distortion for non-standard data product

by gnwiii » Sat Oct 30, 2021 9:35 am America/New_York

yonisherman wrote: Fri Oct 29, 2021 12:34 pm America/New_York Hi,
I just wanted to update on this if anyone in the future comes across a similar issue. As gnwiii commented, if l2bin/l3bin/l3mapgen are working, the problem isn't with the xml file where I added the new product/algorithm. Following gnwiii's suggestion, I used gdalinfo to inspect the netcdf l2 file passed into l2bin and onwards. It appeared that when I appended my custom chl product into the netcdf file I added it as a 64-bit floating-point rather then a 32-bit floating point used for the products produced by l2gen. After changing this the l3maps look normal.

Thanks gnwiii for the help.
Jonathan
You are very welcome -- I like to see more people adding products to level-2 files, as the alternative usually means the binning workflow can't be applied.

Please clarify -- did your level-2 have 64-bit data mis-labelled as 32-bit or was it stored
properly and then mis-handled somewhere in the binning workflow?

yonisherman
Posts: 11
Joined: Wed Mar 27, 2019 4:17 pm America/New_York

Re: l2bin/l3bin/l3mapgen distortion for non-standard data product

by yonisherman » Mon Nov 01, 2021 11:25 am America/New_York

As far as I can tell it was mis-handled in the binning workflow. When I added my new variable to the l2gen produced netcdf file I designated the new variable as a float64 (which I changed to 32-bit the other day) and I guess during binning it was processed as a 32-bit data. I am not entirely sure how and where the binning process decided the data type, and what I can modify in my code so that it would accept the previous 64-bit version. I tried looking in the product.xml and assume it has to do with the "short" data type designation (although I don't see this parameter assigned to the chlor_a section).

Happy this question is leading to some insight on further utilizing the binning workflow.
JS

gnwiii
Posts: 642
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

Re: l2bin/l3bin/l3mapgen distortion for non-standard data product

by gnwiii » Mon Nov 01, 2021 5:16 pm America/New_York

It still isn't clear to me whether the problem is that l2bin mishandled a 64-bit array or you stored a 32-bit array but labelled it as 64-bit data. The latter seems more likely as the dimensions of the array would be smaller by a factor of 2. Remote sensing often uses 32-bit arrays for geophysical data to save space while still providing more precision than most date warrant. Modern computers, however, generally do numerical calculations in 64-bit precision, so extra work may be needed to get a result as 32-bit data. I'll need to do some experiments with different languages -- the original calculation was done soon after netcdf4 support was added to the OCSSW software so libraries and languages have changed.

yonisherman
Posts: 11
Joined: Wed Mar 27, 2019 4:17 pm America/New_York

Re: l2bin/l3bin/l3mapgen distortion for non-standard data product

by yonisherman » Tue Nov 02, 2021 11:51 am America/New_York

It was in fact the latter. It was a 32bit array I labeled as 64.
Thank you for all the help!

Post Reply