how to reduce the size of files generated by l2gen

[zhigang]

Hi all,

I am looking for a solution to reduce the file size when I process the MODIS data using l2gen. As you know, one whole granule of MODIS L2 data including the Rrs at all wavelengths is about 500M or even more, so that a long-time series requires a big space to store them.

I am wondering whether there are some parameters to export the data in the int32 type rather than float numbers in l2gen settings. Any suggestions are appreciated!

Regards,
Zhigang

[gnwiii]

Scanning the NASA L2_LAC_OC.nc files, I don't seen any over 100 MBytes. 

From ncdump -h A2020213212000.L2_LAC_OC.nc (standard NASA level-2 product)

        short Rrs_412(number_of_lines, pixels_per_line) ;
                Rrs_412:long_name = "Remote sensing reflectance at 412 nm" ;
                Rrs_412:scale_factor = 2.e-06f ;
                Rrs_412:add_offset = 0.05f ;
                Rrs_412:units = "sr^-1" ;
                Rrs_412:standard_name = "surface_ratio_of_upwelling_radiance_emerging_from_sea_water_to_downwelling_radiative_flux_in_air" ;
                Rrs_412:_FillValue = -32767s ;
                Rrs_412:valid_min = -30000s ;
                Rrs_412:valid_max = 25000s ;
                Rrs_412:solar_irradiance = 1711.818f ;

NetCDF short is a 16-bit signed integer.   Where are you seeing floats?   NetCDF4-CF can use internal compression, so if you are using HDF4  format, switching to NectCDF4-CF should save space.

[zhigang]

Thanks, George. I did use the HDF5 and the file includes the Rrs, rhos, and view geometries. I will try to use NetCDF.

Thank you, again!

Regards,
Zhigang

[gnwiii]

NetCDF4-CF uses the HDF5 format with some constraints and CF metadata conventions.   HDF5 may be necessary for some types of data that don't fit the constraints and older software that lacks support for NetCDF4-CF.   The linux "file" command may report HDF5 for a NetCDF4-CF file:
% file A2006167181000.L2
A2006167181000.L2: Hierarchical Data Format (version 5) data

Try:
% ncdump -h A2006167181000.L2 | head -1
netcdf A2006167181000 {

l2gen only supports HDF4 and NetCDF4.  From l2gen --help:
  oformat (string) (default=netCDF4) = output file format
        netcdf4: output a netCDF version 4 file
        hdf4:    output a HDF version 4 file
    :Conventions = "CF-1.6 ACDD-1.3" ;

To determine if a file is NetCDF4-CF:
% ncdump -h A2006167181000.L2 | grep ':Conventions'
    :Conventions = "CF-1.6 ACDD-1.3" ;

(See: Attribute Convention for Data Discovery 1.3)

[OB.DAAC - SeanBailey]

The data type for the output products is controlled by the $OCDATAROOT/common/product.xml file.  Rrs and the view geometries are already stored as scaled integers, however rhos is stored as floats.
You can change that by editing the product.xml file entry for the rhos product.  Look for examples in that file...  The more products you add, the larger the file will be, so if space is a consideration fo r you, choose wisely...  Outputting at the 250m or 500m resolution will also increase the file size....using 250m resolution can easily produce a 2GB file with just a dozen or so products.  Data reduction can be had with L3 processing.

Sean

[zhigang]

Hi Sean,

Thanks for your solutions, it is what I want to do. Because I need to generate many products from one whole granule imagery to further process using my programs so that I must consider the disk space in a big ROI.

This XML file works perfectly!:grin:

Thank you again!

Regards,
Zhigang

[zhigang]

Thanks, George! I got a new skill to check out file information in the shell terminal. - Zhigang