Thousands of GPM IMERG files generated when downloading 30min data
-
jared_kodero
- Posts: 2
- Joined: Sun Sep 10, 2023 8:43 am America/New_York
Thousands of GPM IMERG files generated when downloading 30min data
I am downloading gpm IMERG files for Boston metro starting 2000 to date. However, there are thousands of links > 300,000, which, when I download, generates 300,000.nc4 files each being 26kb ( I am guessing 30mins of data) if there a way to combine all the data and get 1 nc4 file before download? The thousands of files are really hard to process.
Tags:
-
GES DISC - mgreene
- User Services
- Posts: 198
- Joined: Wed Sep 25, 2019 9:35 am America/New_York
- Been thanked: 1 time
Re: Thousands of GPM IMERG files generated when downloading 30min data
Thank you for your question. A Subject Matter Expert has been notified and will respond to your question shortly.
Re: Thousands of GPM IMERG files generated when downloading 30min data
Hello,
Thank you for reaching out to us!
Before we provide some strategies, I do want to caution that subsetting granules en masse (usually over a year's worth) may cause rate limiting on your Earthdata account. If you do decide to subset large amounts of granules at once, please be judicious in the chunks of time that you select and how often you select them, and that there may be errors or interruptions.
One way to do this is by aggregating along the time dimension using a custom range with the TDS aggregator tool (https://disc.gsfc.nasa.gov/information/howto?keywords=THREDDS&title=How%20to%20Obtain%20a%20Spatially%20Subsetted%20Time%20Series%20using%20TDS%20). You can find the appropriate GPM TDS server from this list: https://disc.gsfc.nasa.gov/information/tools?title=OPeNDAP%20and%20GDS.
You can also access that aggregate file using Python and "Xarray", by following the "THREDDS in Xarray" step in this how-to: https://disc.gsfc.nasa.gov/information/howto?keywords=python&title=How%20to%20Access%20GES%20DISC%20Data%20Using%20Python. This allows you to subset the aggregated file as a single Xarray dataset.
Another option is to use NCO to concatenate along the time dimension, by following this how-to: https://disc.gsfc.nasa.gov/information/howto?keywords=nco&title=How%20to%20Concatenate%20the%20Time%20Dimension%20of%20netCDF%20Files%20with%20NCO.
Please let us know if this helps guide you in the right direction, and thank you for contacting us!
Chris Battisto
Thank you for reaching out to us!
Before we provide some strategies, I do want to caution that subsetting granules en masse (usually over a year's worth) may cause rate limiting on your Earthdata account. If you do decide to subset large amounts of granules at once, please be judicious in the chunks of time that you select and how often you select them, and that there may be errors or interruptions.
One way to do this is by aggregating along the time dimension using a custom range with the TDS aggregator tool (https://disc.gsfc.nasa.gov/information/howto?keywords=THREDDS&title=How%20to%20Obtain%20a%20Spatially%20Subsetted%20Time%20Series%20using%20TDS%20). You can find the appropriate GPM TDS server from this list: https://disc.gsfc.nasa.gov/information/tools?title=OPeNDAP%20and%20GDS.
You can also access that aggregate file using Python and "Xarray", by following the "THREDDS in Xarray" step in this how-to: https://disc.gsfc.nasa.gov/information/howto?keywords=python&title=How%20to%20Access%20GES%20DISC%20Data%20Using%20Python. This allows you to subset the aggregated file as a single Xarray dataset.
Another option is to use NCO to concatenate along the time dimension, by following this how-to: https://disc.gsfc.nasa.gov/information/howto?keywords=nco&title=How%20to%20Concatenate%20the%20Time%20Dimension%20of%20netCDF%20Files%20with%20NCO.
Please let us know if this helps guide you in the right direction, and thank you for contacting us!
Chris Battisto