Earthdata Forum

Hi all,
I am trying to test an MDN approach to retrieve chl a using MODIS Rrs (see Pahlevan et al. 2020, https://www.sciencedirect.com/science/a ... 5719306248).
I am using L2 data files I created using the standard l2gen (at 1000m resolution). Afterwards, I use the Rrs in the model and append the result in the L2 nc file as chl_mdn. In order to proceed with l2bin-l3bin-l3mapgen to get weekly composites I first added a parameter in the product.xml file, following a forum post I saw. When I plot the l2 data directly the data looks good and I can compare with the standard chl a product, however the chl_mdn mapped image is completely distorted (see attache image). I am not sure if I am using any of the steps incorrectly (standard parameters for all steps at 1km and 'regional' prodtype) , or perhaps my input in product.xml is incorrect (wasn't sure if need to be under the overarching chl product as an algorithm or as a separate product, currently I am using the latter, see below).

Do you have any suggestions about using non-standard products as far as binning goes.
Thank you so much
Jonathan Sherman


</product>
<product name="chl_mdn">
<standardName>mass_concentration_of_chlorophyll_in_sea_water dervied using MDN</standardName>
<units>mg m^-3</units>
<category>Derived</category>
<range>
<validMin>0.001</validMin>
<validMax>100</validMax>
<displayMin>0.01</displayMin>
<displayMax>20</displayMax>
</range>
<algorithm>
<cat_ix>-1</cat_ix><!-- The cat_ix is required by l2gen - Since you're not creating the product within l2gen, leave this alone -->
<description>Chl from MDN without 748nm</description>
<reference>Pahlevan, N., B. Smith, J. Schalles, et al. 2020. "Seamless retrievals of chlorophyll-a from Sentinel-2 (MSI) and Sentinel-3 (OLCI) in inland and coastal waters: A machine-learning approach." Remote Sensing of Environment, 111604 [10.1016/j.rse.2019.111604]</reference>
</algorithm>
</product>

A number of years ago I was able to add a non-standard product to NASA level-2 files and did not encounter problems with mapping. My guess is that the xml file is not the cause. Looking at the NetCDF header with "ncdump -h" for differences between NASA chl variables and the new variable might be informative. You can ary using gdalinfo to compare the mapping between a NASA chl and your product.

Thank you for the reply. I am pretty sure I had a used the get_product_info command after editing the product.xml file to see if it was added, but I should double check to see how it fully compares with the standard chla product.

yonisherman wrote: ↑Wed Oct 27, 2021 10:40 pm America/New_York Thank you for the reply. I am pretty sure I had a used the get_product_info command after editing the product.xml file to see if it was added, but I should double check to see how it fully compares with the standard chla product.

If the .xml is bad, the binning and mapping would complain, so I suspect the problem is with the coordinates. I guess you are using Panoply, so you might see a difference in the metadata for the level-2 file in Panoply's metadata display. You can also use l3bindump to inspect a few test locations. I expect you will find problems in the binned files.

Hi,
I just wanted to update on this if anyone in the future comes across a similar issue. As gnwiii commented, if l2bin/l3bin/l3mapgen are working, the problem isn't with the xml file where I added the new product/algorithm. Following gnwiii's suggestion, I used gdalinfo to inspect the netcdf l2 file passed into l2bin and onwards. It appeared that when I appended my custom chl product into the netcdf file I added it as a 64-bit floating-point rather then a 32-bit floating point used for the products produced by l2gen. After changing this the l3maps look normal.

Thanks gnwiii for the help.
Jonathan

yonisherman wrote: ↑Fri Oct 29, 2021 12:34 pm America/New_York Hi,
I just wanted to update on this if anyone in the future comes across a similar issue. As gnwiii commented, if l2bin/l3bin/l3mapgen are working, the problem isn't with the xml file where I added the new product/algorithm. Following gnwiii's suggestion, I used gdalinfo to inspect the netcdf l2 file passed into l2bin and onwards. It appeared that when I appended my custom chl product into the netcdf file I added it as a 64-bit floating-point rather then a 32-bit floating point used for the products produced by l2gen. After changing this the l3maps look normal.

Thanks gnwiii for the help.
Jonathan

You are very welcome -- I like to see more people adding products to level-2 files, as the alternative usually means the binning workflow can't be applied.

Please clarify -- did your level-2 have 64-bit data mis-labelled as 32-bit or was it stored
properly and then mis-handled somewhere in the binning workflow?

As far as I can tell it was mis-handled in the binning workflow. When I added my new variable to the l2gen produced netcdf file I designated the new variable as a float64 (which I changed to 32-bit the other day) and I guess during binning it was processed as a 32-bit data. I am not entirely sure how and where the binning process decided the data type, and what I can modify in my code so that it would accept the previous 64-bit version. I tried looking in the product.xml and assume it has to do with the "short" data type designation (although I don't see this parameter assigned to the chlor_a section).

Happy this question is leading to some insight on further utilizing the binning workflow.
JS

It still isn't clear to me whether the problem is that l2bin mishandled a 64-bit array or you stored a 32-bit array but labelled it as 64-bit data. The latter seems more likely as the dimensions of the array would be smaller by a factor of 2. Remote sensing often uses 32-bit arrays for geophysical data to save space while still providing more precision than most date warrant. Modern computers, however, generally do numerical calculations in 64-bit precision, so extra work may be needed to get a result as 32-bit data. I'll need to do some experiments with different languages -- the original calculation was done soon after netcdf4 support was added to the OCSSW software so libraries and languages have changed.

It was in fact the latter. It was a 32bit array I labeled as 64.
Thank you for all the help!