how to reduce the size of files generated by l2gen

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
zhigang
Posts: 74
Joined: Tue Nov 10, 2020 8:03 pm America/New_York
Answers: 0

how to reduce the size of files generated by l2gen

by zhigang » Wed Aug 05, 2020 2:33 pm America/New_York

Hi all,

I am looking for a solution to reduce the file size when I process the MODIS data using l2gen. As you know, one whole granule of MODIS L2 data including the Rrs at all wavelengths is about 500M or even more, so that a long-time series requires a big space to store them.

I am wondering whether there are some parameters to export the data in the int32 type rather than float numbers in l2gen settings. Any suggestions are appreciated!

Regards,
Zhigang

Filters:

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

how to reduce the size of files generated by l2gen

by gnwiii » Wed Aug 05, 2020 3:13 pm America/New_York

Scanning the NASA L2_LAC_OC.nc files, I don't seen any over 100 MBytes. 

From ncdump -h A2020213212000.L2_LAC_OC.nc (standard NASA level-2 product)

        short Rrs_412(number_of_lines, pixels_per_line) ;
                Rrs_412:long_name = "Remote sensing reflectance at 412 nm" ;
                Rrs_412:scale_factor = 2.e-06f ;
                Rrs_412:add_offset = 0.05f ;
                Rrs_412:units = "sr^-1" ;
                Rrs_412:standard_name = "surface_ratio_of_upwelling_radiance_emerging_from_sea_water_to_downwelling_radiative_flux_in_air" ;
                Rrs_412:_FillValue = -32767s ;
                Rrs_412:valid_min = -30000s ;
                Rrs_412:valid_max = 25000s ;
                Rrs_412:solar_irradiance = 1711.818f ;


NetCDF short is a 16-bit signed integer.   Where are you seeing floats?   NetCDF4-CF can use internal compression, so if you are using HDF4  format, switching to NectCDF4-CF should save space.

zhigang
Posts: 74
Joined: Tue Nov 10, 2020 8:03 pm America/New_York
Answers: 0

how to reduce the size of files generated by l2gen

by zhigang » Wed Aug 05, 2020 11:16 pm America/New_York

Thanks, George. I did use the HDF5 and the file includes the Rrs, rhos, and view geometries. I will try to use NetCDF.

Thank you, again!

Regards,
Zhigang

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

how to reduce the size of files generated by l2gen

by gnwiii » Thu Aug 06, 2020 7:34 am America/New_York

NetCDF4-CF uses the HDF5 format with some constraints and CF metadata conventions.   HDF5 may be necessary for some types of data that don't fit the constraints and older software that lacks support for NetCDF4-CF.   The linux "file" command may report HDF5 for a NetCDF4-CF file:
% file A2006167181000.L2
A2006167181000.L2: Hierarchical Data Format (version 5) data

Try:
% ncdump -h A2006167181000.L2 | head -1
netcdf A2006167181000 {

l2gen only supports HDF4 and NetCDF4.  From l2gen --help:
  oformat (string) (default=netCDF4) = output file format
        netcdf4: output a netCDF version 4 file
        hdf4:    output a HDF version 4 file
    :Conventions = "CF-1.6 ACDD-1.3" ;

To determine if a file is NetCDF4-CF:
% ncdump -h A2006167181000.L2 | grep ':Conventions'
    :Conventions = "CF-1.6 ACDD-1.3" ;

(See: Attribute Convention for Data Discovery 1.3)

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1484
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

how to reduce the size of files generated by l2gen

by OB.DAAC - SeanBailey » Sat Aug 08, 2020 11:18 am America/New_York

The data type for the output products is controlled by the $OCDATAROOT/common/product.xml file.  Rrs and the view geometries are already stored as scaled integers, however rhos is stored as floats.
You can change that by editing the product.xml file entry for the rhos product.  Look for examples in that file...  The more products you add, the larger the file will be, so if space is a consideration fo r you, choose wisely...  Outputting at the 250m or 500m resolution will also increase the file size....using 250m resolution can easily produce a 2GB file with just a dozen or so products.  Data reduction can be had with L3 processing.

Sean

zhigang
Posts: 74
Joined: Tue Nov 10, 2020 8:03 pm America/New_York
Answers: 0

how to reduce the size of files generated by l2gen

by zhigang » Sat Aug 08, 2020 2:29 pm America/New_York

Hi Sean,

Thanks for your solutions, it is what I want to do. Because I need to generate many products from one whole granule imagery to further process using my programs so that I must consider the disk space in a big ROI.

This XML file works perfectly!:grin:

Thank you again!

Regards,
Zhigang

zhigang
Posts: 74
Joined: Tue Nov 10, 2020 8:03 pm America/New_York
Answers: 0

how to reduce the size of files generated by l2gen

by zhigang » Sat Aug 08, 2020 2:33 pm America/New_York

Thanks, George! I got a new skill to check out file information in the shell terminal. - Zhigang

Post Reply