Thousands of GPM IMERG files generated when downloading 30min data

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
jared_kodero
Posts: 2
Joined: Sun Sep 10, 2023 8:43 am America/New_York
Answers: 0

Thousands of GPM IMERG files generated when downloading 30min data

by jared_kodero » Sun Sep 10, 2023 9:11 am America/New_York

I am downloading gpm IMERG files for Boston metro starting 2000 to date. However, there are thousands of links > 300,000, which, when I download, generates 300,000.nc4 files each being 26kb ( I am guessing 30mins of data) if there a way to combine all the data and get 1 nc4 file before download? The thousands of files are really hard to process.

Tags:

GES DISC - mgreene
User Services
User Services
Posts: 198
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Thousands of GPM IMERG files generated when downloading 30min data

by GES DISC - mgreene » Mon Sep 11, 2023 9:17 am America/New_York

Thank you for your question. A Subject Matter Expert has been notified and will respond to your question shortly.

cbattisto
Posts: 13
Joined: Wed Feb 16, 2022 4:38 pm America/New_York
Answers: 0

Re: Thousands of GPM IMERG files generated when downloading 30min data

by cbattisto » Tue Sep 12, 2023 2:16 pm America/New_York

Hello,

Thank you for reaching out to us!

Before we provide some strategies, I do want to caution that subsetting granules en masse (usually over a year's worth) may cause rate limiting on your Earthdata account. If you do decide to subset large amounts of granules at once, please be judicious in the chunks of time that you select and how often you select them, and that there may be errors or interruptions.

One way to do this is by aggregating along the time dimension using a custom range with the TDS aggregator tool (https://disc.gsfc.nasa.gov/information/howto?keywords=THREDDS&title=How%20to%20Obtain%20a%20Spatially%20Subsetted%20Time%20Series%20using%20TDS%20). You can find the appropriate GPM TDS server from this list: https://disc.gsfc.nasa.gov/information/tools?title=OPeNDAP%20and%20GDS.

You can also access that aggregate file using Python and "Xarray", by following the "THREDDS in Xarray" step in this how-to: https://disc.gsfc.nasa.gov/information/howto?keywords=python&title=How%20to%20Access%20GES%20DISC%20Data%20Using%20Python. This allows you to subset the aggregated file as a single Xarray dataset.

Another option is to use NCO to concatenate along the time dimension, by following this how-to: https://disc.gsfc.nasa.gov/information/howto?keywords=nco&title=How%20to%20Concatenate%20the%20Time%20Dimension%20of%20netCDF%20Files%20with%20NCO.

Please let us know if this helps guide you in the right direction, and thank you for contacting us!

Chris Battisto

Post Reply