MODIS Aqua POC 2002-2014 invalid values

[jason.roberts]

Hi OceanColor team,

I noticed that the MODIS Aqua POC 9km annual composites from 2002-2013 use float32 as their data type with no scaling equation, but the values range from roughly -2150 to 2150. These appear to be invalid. By contrast, the files from 2014-2016 use int16 with a scaling equation; when applied the values come out in an expected range, e.g. for 2016 they range from 17 to 12953.

Here is an example file, if you want to examine it: https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A20130012013365.L3m_YR_POC_poc_9km.nc

I suspect there was a snafu with the 2014.0 or 2014.1 reprocessing that converted everything to netCDF4. I noticed there are some other products (e.g. PAR) that appear similar: they were reprocessed into float for the 2002-2014ish period but continue to be produced as int16 with a scaling equation beyond some point in 2014. This is odd but can be dealt with, so long as the consuming application does not assume the entire time series uses the same data type and scaling, but instead checks each file. But POC appears to be broken for the pre-2014 period in that the float32 values appear bogus.

Could you please take a look at this and advise me on what to do? It would be great if you could reprocess the POC files that are broken, but some other workaround would be fine. Thank you very much for maintaining the quality of your archive; it has always served me and my colleagues well over the years and we really appreciate it.

I have not looked in detail at other averagings (8 day, monthly, etc.) or 4 km resolution, but I would guess this problem exists with them as well.

Finally, I'm sure you know this, but I am reporting a different issue than this post https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=7237. Just wanted to be clear about that.

Thanks again,
Jason

[jason.roberts]

Ok, so I was a bit hasty in my investigation. Apologies for that.

What it actually looks like is that the 2002-2014 and 2014-2016 periods have similar values in many places throughout the ocean, but the 2002-2014 period shows large negative values in areas of high POC, e.g. in estuaries or close to shore in the Northwest African upwelling. This is definitely not as problematic as if all the values were wrong. But it would be great if this problem could be corrected, as it limits the utility of this product.

Thanks,
Jason

[OB.DAAC - SeanBailey]

The product includes the "valid_min" and "valid_max" attributes.  The data generated with the older smigen code were reported as floats, and the calculation could go negative - but that doesn't mean it is valid :wink:

The newer l3mapgen generated products are scaled - and as such the negative values are "scaled away".  There is no inconsistency if you used the valid_min/max attributes.

Regards,
Sean

[gnwiii]

The conventions used for NetCDF4-CF include scaling, missing value support, and valid min and max values.   In principle, it is possible to have a data set in every file uses different internal representation and scaling.  In practice, many people are still using legacy software that doesn't fully support NetCDF4-CF where these details have to handled manually.   If you are familiar with the HDF Group's "zoo" of hdf4 examples, you can appreciate that converting these examples to work with current NetCDF4-CF products gives much simpler files because the library handles these details "automagically".

[simonf]

Hi,

I'm still seeing negative poc - eg, in https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2003004.L3m_DAY_POC_poc_4km.nc
This file is produced with l3mapgen, and scale/offset are 1/0, so the earlier explanation about bad values produced by smigen don't seem applicable.

Do you know how we should interpret such values?

Thanks,
Simon

[OB SeaDAS - dshea]

The values in that file are stored as a scaled shorts.  You need to apply scale_factor and add_offset to get the POC in geophysical units.

In your example file: A2003004.L3m_DAY_POC_poc_4km.nc
poc:scale_factor = 0.2f    <- not 1.0
poc:add_offset = 6400.f  <- not 0.0
min file value = 8.0
max file value = 12953.4

so, I don't see any negative values.  Of course many of the unconverted short values are negative.

don

[simonf]

Hm, how are you reading the scale/offset metadata? Both gdalinfo and ncdump show me this:
       poc:scale_factor = 1.f ;
       poc:add_offset = 0.f ;

[OB.DAAC - SeanBailey]

Seems you have an old version of the file.
The current version has:  :date_created = "2017-12-29T22:22:57.000Z"
...but I think you may have figured that out, given your subsequent post...

Sean

[lagbleze]

Hello OceanColor Team,

I would like to know if the dataset "NASA/OCEANDATA/MODIS-Aqua/L3SMI" comes with cloud already removed?

I am trying to do a Time series analysis of chlorophyll, POC and SST  in Earth Engine and was wondering if I have to first do a cloud removal or other other preprocessing analysis.

Thank you.

[OB.DAAC - SeanBailey]

Level 3 products have a number of masks applied to ensure the highest quality data....yes, we exclude clouds.