Merra2 Data Downloading

[yukee121]

goldsmr5.gesdisc.eosdis.nasa.gov
https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NVASM.5.12.4

Hi, I just want to ask why downloading this 3D file of Merra2 is so slow. I use wget method to download which spends 8 minutes to finish downloading and the downloading speed is less than 5MB/second. But downloading speed of other files such as https://goldsmr4.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I1NXASM.5.12.4 is quite fast, which has over more than 20MB/second. If there are any suggestions why this happens, it will be really helpful. Thank you very much!

[GES DISC - xpan2]

Thanks for reaching out, and apologies for the inconvenience. Could you please share the wget command you’re using along with the data URL? That will help us identify which data service is causing the issue. Thank you.

NASA GES DISC Help Desk

[yukee121]

def download_merra_file(
url: str,
destination: Union[str, Path],
force: bool = False,
credentials: Optional[Tuple[str, str]] = None
) -> Path:
"""
Download MERRA2 file if it does not already exist.

Args:
url: String containing the URL of the file to download.
destination: The folder to which to download the file.
force: Set to True to force download even if file exists locally.
credentials: Unused here. Authentication is expected through ~/.urs_cookies.

Return:
A Path object pointing to the local file.
"""
destination = Path(destination)
destination.mkdir(exist_ok=True, parents=True)

filename = url.split("/")[-1]
file_path = destination / filename

if not force and file_path.exists():
try:
data = xr.open_dataset(file_path)
data.close()
return file_path
except Exception:
file_path.unlink()

cookie_file = Path("~/.urs_cookies").expanduser()

if not cookie_file.exists():
raise FileNotFoundError(
f"Cookie file not found: {cookie_file}. "
"Please create ~/.urs_cookies first."
)

cmd = [
"wget",
"--load-cookies", str(cookie_file),
"--save-cookies", str(cookie_file),
"--keep-session-cookies",
"--content-disposition",
"-O", str(file_path),
url,
]
try:
subprocess.run(cmd, check=True)
except subprocess.CalledProcessError as e:
if file_path.exists():
file_path.unlink()
raise RuntimeError(f"wget download failed for {url}") from e
return file_path


Inside this function, cmd is how I use wget to download the data. Thank you for helping me!

[GES DISC - xpan2]

Thanks for sharing the code. I actually asked the "url: String containing the URL of the file to download". I assume you have downloaded many files. I just need an example of one file to get the idea where you get the data.

[yukee121]

wget \
--load-cookies ~/.urs_cookies \
--save-cookies ~/.urs_cookies \
--keep-session-cookies \
--content-disposition \
-O ./MERRA2_100.inst3_3d_asm_Nv.19800101.nc4 \
"https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NVASM.5.12.4/1980/01/MERRA2_100.inst3_3d_asm_Nv.19800101.nc4"


especially all of this 3d data file is downloading slow, needs around 10 minutes to finish downloading one data. url: https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NVASM.5.12.4/1980/01/MERRA2_100.inst3_3d_asm_Nv.19800101.nc4

[GES DISC - xpan2]

GES DISC has introduced a new service to help users download data more efficiently. You may want to check it out to see if it improves your download speed. It also offers an option to generate Python code for downloading data.

First, go to the dataset landing page:
https://disc.gsfc.nasa.gov/datasets/M2I3NVASM_5.12.4/summary?keywords=M2I3NVASM_5.12.4

Then click the green “Beta” button under “Data Access.” From there, you can choose either:

“Get Original Files” (no subsetting), or
“Subset Data” (by spatial coverage or variables)

Follow the steps provided in the interface to complete your request.

Let me know how it works for you.

[yukee121]

Thank you for replying, I see that option, but I did not find the python scripts to do download, can you share me with a link or example for downloading the subset of data? Thanks

[GES DISC - xpan2]

Please see the highlighted part in screenshot. You have to make all selections and then click on "Download Option" to download "Python Script".