Download L3SMI data using R

[ygrund]

Hi there,

I would like to modify the R script of "DAAC_Data_Download_R" (https://git.earthdata.nasa.gov/projects/LPDUR/repos/daac_data_download_r/browse/DAACDataDownload.R) to download L3SMI data located at https://oceandata.sci.gsfc.nasa.gov (directaccess/CYAN/L3SMI/). With my Earthdata account credentials, I had no problem to download the .hdf file with the example link in the script, but I couldn't download the .tif files by simply replacing the the link in the script as following,
files <- "https://oceandata.sci.gsfc.nasa.gov (/directaccess/CYAN/L3SMI/2020/366/L2020366.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif)"

The error message is
"
Downloading: 460 B [1] "L2020366.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif not downloaded. Verify that your username and password are correct in C:\\Users\\ygrund/.netrc"
"

I am not sure if the error was truly caused by the credentials. It would be very helpful if you could provide me any suggestions.

Thank you,
Yuan Grund

[gnwiii]

The R script "works for me" using <<files <- "https://oceandata.sci.gsfc.nasa.gov (/cgi/getfile/L2020366.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif)">>. You can generate a list of URL's using the file search page: [url]https://oceandata.sci.gsfc.nasa.gov (/api/file_search/)[/url]. In the past, I found I sometimes needed two entries in my .netrc file, one for "machine urs.earthdata.nasa.gov" and one for "machine oceandata.sci.gsfc.nasa.gov".

[ygrund]

Thanks, ggwii. No I didn't have the luck to get the file downloaded after I wrote both lines "machine urs.earthdata.nasa.gov" and "machine oceandata.sci.gsfc.nasa.gov" to netrc_conn. I attached the R script in this request. Appreciate if you could take a look if there is anything I missed. Thanks!

[OB.DAACx - amscott]

I see a message from the Earthdata helpdesk at the bottom of your script that points to potential issues you should investigate and try to correct. Have you addressed them already?

###
# Issue: ---
# Author: Catalino Cuadrado (catalino.cuadrado@nasa.gov) # Date: 12-19-2018
################################################################################# # Set up R
# You may need to install the httr package.
# install.packages("httr") setwd("C:/Users/ygrund/ygrund_C_Users/PROJECTS/HABs/Branches/2022/ HAB_Shiny_app")
library(httr)
netrc_path <- "./Earthdata/.netrc"
cookie_path <- "./Earthdata/.urs_cookies"
downloaded_file_path <- "./Earthdata/2020/ L2020301.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif"
# Before using the script
#Set up your ~/.netrc file as listed here: https://wiki.earthdata.nasa.gov/ display/EL/How+To+Access+Data+With+cURL+And+Wget set_config(config(followlocation=1,netrc=1,netrc_file=netrc_path,cookie=cookie_path,cookie httr::GET(url = "https://oceandata.sci.gsfc.nasa.gov (/directaccess/CYAN/L3SMI/ 2020/301/L2020301.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif)",
write_disk(downloaded_file_path, overwrite = TRUE))

From that message, you can see that your .netrc file is not where it should be. Your 'home' directory is directly under your user. This path appears to be "C:/Users/ygrund/". Please work through the notes shared by the helpdesk above and try your download again.

[ygrund]

Hello again, I've returned to the issue of downloading the data. I've noticed that the file directory has been updated. For instance, the link for the file I need to download using R is now: 'https://oceandata.sci.gsfc.nasa.gov/getfile/L2023001.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif'.

I've set up the .netrc and .urs_cookies as per Catalio Cuadrado's instructions (https://wiki.earthdata.nasa.gov/display/EL/How+to+access+data+with+R). The file size should be 136KB, but when using the following code, the downloaded file only has a size of 10KB, indicating an unsuccessful download. Please see my codes below. Any suggestions you could offer would be greatly appreciated.

library(httr)
netrc_path <- "C:/Users/ygrund/ygrund_C_Users/PROJECTS/HABs/HABShiny_Oceandata/.netrc"
cookie_path <- "C:/Users/ygrund/ygrund_C_Users/PROJECTS/HABs/HABShiny_Oceandata/.urs_cookies"
downloaded_file_path <- "C:/Users/ygrund/ygrund_C_Users/PROJECTS/HABs/HABShiny_Oceandata/data/1.tif"

httr::set_config(config(followlocation=1,netrc=1,netrc_file=netrc_path,cookie=cookie_path,cookiefile=cookie_path,cookiejar=cookie_path))

httr::GET(url = "https://oceandata.sci.gsfc.nasa.gov/getfile/L2023001.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif", write_disk(downloaded_file_path, overwrite = TRUE))

[OB.DAACx - SeanBailey]

The 10kb file is probably the HTML page for the login redirect. This means something isn't right with the way your credentials are passed, and you were not logged into your EDL account. Verify that your .netrc file is correctly defined.

Sean

[ygrund]

Hi Sean,

Thank you for your prompt response and for addressing the issue regarding the login redirect. I believe that it might be a problem with the credentials. After several attempts, I have been unable to resolve the issue with the .netrc file while using Catalio Cuadrado's script. I am considering the possibility that it could be related to the IT network configuration within my organization.

However, I have had success in converting the Python script available on the EarthData User website (https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+Python) into an R script, and it works perfectly! :-) In case others encounter a similar problem, I am sharing my script here for reference.

library(httr)

# Function to create a custom session for authentication
session <- function(username, password) {
httr::authenticate(username, password)
}

# Define username and password
username <- "xxxx"
password <- "xxxx"

# Create the session
my_session <- session(username, password)

# URL of the file to retrieve
url <- "https://oceandata.sci.gsfc.nasa.gov/getfile/L2023201.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_2_2.tif"

# Extract the filename from the URL
filename <- sub(".*/", "", url)

tryCatch({
# Submit the request using the configured session
response <- httr::GET(
url,
httr::progress(),
httr::add_headers(`User-Agent` = "Mozilla/5.0"),
httr::authenticate(username, password)
)

# Check for successful response
if (httr::status_code(response) == 200) {
# Save the file
content <- httr::content(response, as = "raw")
writeBin(content, filename)
print("File downloaded successfully")
} else {
stop("Failed to download the file")
}
}, error = function(e) {
# Handle any errors here
print(paste("Error:", e$message))
})

Thank you again for your assistance.
Yuan