R Script for downloading MODIS data

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
andystheo
Posts: 1
Joined: Thu Jul 21, 2022 1:38 pm America/New_York
Answers: 0

R Script for downloading MODIS data

by andystheo » Thu Jul 21, 2022 1:42 pm America/New_York

Hello everyone,

I am having issues downloading the MODIS data from R. I am using the R script below. I get two errors. The first is that the connection is invalid. This happens when I enter my credentials. The second is when I run the loop to download everything which says that there was an SSL connection timeout.

Any tips on how to resolve this or perhaps alternative ways to bulk download data from MODIS?






# ------------------------------------------------------------------------------------------------ #
# How to Access the LP DAAC Data Pool with R
# The following R code example demonstrates how to configure a connection to download data from an
# Earthdata Login enabled server, specifically the LP DAAC Data Pool.
# ------------------------------------------------------------------------------------------------ #
# Author: LP DAAC
# Last Updated: 06/07/2022
# ------------------------------------------------------------------------------------------------ #
# Check for required packages, install if not previously installed
if ("sys" %in% rownames(installed.packages()) == FALSE) {install.packages("sys")}
if ("getPass" %in% rownames(installed.packages()) == FALSE) { install.packages("getPass")}
if ("httr" %in% rownames(installed.packages()) == FALSE) { install.packages("httr")}

# Load necessary packages into R
library(sys)
library(getPass)
library(httr)
# ---------------------------------SET UP ENVIRONMENT--------------------------------------------- #
# IMPORTANT: Update the line below if you want to download to a different directory (ex: "c:/data/")
dl_dir <- Sys.getenv("HOME") # Set dir to download files to
setwd(dl_dir) # Set the working dir to the dl_dir
usr <- file.path(Sys.getenv("USERPROFILE")) # Retrieve home dir (for netrc file)
if (usr == "") {usr = Sys.getenv("HOME")} # If no user profile exists, use home
netrc <- file.path(usr,'.netrc', fsep = .Platform$file.sep) # Path to netrc file

# ------------------------------------CREATE .NETRC FILE------------------------------------------ #
# If you already have a .netrc file with your Earthdata Login credentials stored in your home
# directory, this portion will be skipped. Otherwise you will be prompted for your NASA Earthdata
# Login Username/Password and a netrc file will be created to store your credentials (in home dir)
if (file.exists(netrc) == FALSE || grepl("urs.earthdata.nasa.gov", readLines(netrc)) == FALSE) {
netrc_conn <- file(netrc)

# User will be prompted for NASA Earthdata Login Username and Password below
writeLines(c("machine urs.earthdata.nasa.gov",
sprintf("login %s", getPass(msg = "Enter NASA Earthdata Login Username \n (or create an account at urs.earthdata.nasa.gov) :")),
sprintf("password %s", getPass(msg = "Enter NASA Earthdata Login Password:"))), netrc_conn)
close(netrc_conn)
}

# ---------------------------CONNECT TO DATA POOL AND DOWNLOAD FILES------------------------------ #
# Below, define either a single link to a file for download, a list of links, or a text file
# containing links to the desired files to download. For a text file, there should be 1 file link
# listed per line. Here we show examples of each of the three ways to download files.
# **IMPORTANT: be sure to update the links for the specific files you are interested in downloading.

# How to get the links:
# 1) go to https://search.earthdata.nasa.gov/search
# 2) select the product, time range, and area you want to download
# 3) click "Download all"
# 4) insert EarthData login credentials if necessary
# 5) click "Download data"
# 6) save list of files in text file -> insert text file location below
# 7) proceed with option 3 ("Text file containing links") below

# 1. Single file (this is just an example link, replace with your desired file to download):
#files <- "https://e4ftl01.cr.usgs.gov/MOLA/MYD09GA.061/2002.07.06/MYD09GA.A2002187.h10v04.061.2020071193416.hdf"

# 2. List of files (these are just example links, replace with your desired files to download:
#files <- c("https://e4ftl01.cr.usgs.gov/MOLA/MYD09GA.061/2002.07.06/MYD09GA.A2002187.h10v04.061.2020071193416.hdf",
# "https://e4ftl01.cr.usgs.gov/MOLT/MOD11A1.061/2000.03.09/MOD11A1.A2000069.h00v08.061.2020043121122.hdf")

# 3. Text file containing links (just an example, replace with your text file location):
files <- readLines("~/Downloads/3301324694-download-2.txt", warn = FALSE)

# Loop through all files
for (i in 1:length(files)) {
filename <- tail(strsplit(files[i], '/')[[1]], n = 1) # Keep original filename

# Write file to disk (authenticating with netrc) using the current directory/filename
response <- GET(files[i], write_disk(filename, overwrite = TRUE), progress(),
config(netrc = TRUE, netrc_file = netrc), set_cookies("LC" = "cookies"))

# Check to see if file downloaded correctly
if (response$status_code == 200) {
print(sprintf("%s downloaded at %s", filename, dl_dir))
} else {
print(sprintf("%s not downloaded. Verify that your username and password are correct in %s", filename, netrc))
}
}

Tags:

LP DAAC - jwilson
User Services
User Services
Posts: 268
Joined: Mon Sep 30, 2019 12:39 pm America/New_York
Answers: 1
Has thanked: 9 times

Re: R Script for downloading MODIS data

by LP DAAC - jwilson » Tue Jul 26, 2022 8:42 am America/New_York

Early last week the USGS EROS Center completed a significant maintenance activity, after bringing the systems back up the LP DAAC is experiencing high volumes of user traffic. This is impacting data downloads for some users and causing timeouts. We are looking into performance improvements affecting data access.

We apologize for any inconvenience this may be causing. Please contact LP DAAC User Services with any questions or concerns.

Post Reply