Download L3SMI data using R

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
ygrund
Posts: 4
Joined: Fri Jan 07, 2022 11:18 am America/New_York
Answers: 0

Download L3SMI data using R

by ygrund » Fri Jan 07, 2022 12:51 pm America/New_York

Hi there,

I would like to modify the R script of "DAAC_Data_Download_R" (https://git.earthdata.nasa.gov/projects/LPDUR/repos/daac_data_download_r/browse/DAACDataDownload.R) to download L3SMI data located at https://oceandata.sci.gsfc.nasa.gov (directaccess/CYAN/L3SMI/). With my Earthdata account credentials, I had no problem to download the .hdf file with the example link in the script, but I couldn't download the .tif files by simply replacing the the link in the script as following,
files <- "https://oceandata.sci.gsfc.nasa.gov (/directaccess/CYAN/L3SMI/2020/366/L2020366.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif)"

The error message is
"
Downloading: 460 B [1] "L2020366.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif not downloaded. Verify that your username and password are correct in C:\\Users\\ygrund/.netrc"
"

I am not sure if the error was truly caused by the credentials. It would be very helpful if you could provide me any suggestions.

Thank you,
Yuan Grund

Tags:

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

Re: Download L3SMI data using R

by gnwiii » Wed Jan 12, 2022 8:36 pm America/New_York

The R script "works for me" using <<files <- "https://oceandata.sci.gsfc.nasa.gov (/cgi/getfile/L2020366.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif)">>. You can generate a list of URL's using the file search page: [url]https://oceandata.sci.gsfc.nasa.gov (/api/file_search/)[/url]. In the past, I found I sometimes needed two entries in my .netrc file, one for "machine urs.earthdata.nasa.gov" and one for "machine oceandata.sci.gsfc.nasa.gov".

ygrund
Posts: 4
Joined: Fri Jan 07, 2022 11:18 am America/New_York
Answers: 0

Re: Download L3SMI data using R

by ygrund » Wed Jan 12, 2022 9:17 pm America/New_York

Thanks, ggwii. No I didn't have the luck to get the file downloaded after I wrote both lines "machine urs.earthdata.nasa.gov" and "machine oceandata.sci.gsfc.nasa.gov" to netrc_conn. I attached the R script in this request. Appreciate if you could take a look if there is anything I missed. Thanks!
Attachments
script.pdf
(48.99 KiB) Downloaded 346 times

OB.DAAC - amscott
User Services
User Services
Posts: 345
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Download L3SMI data using R

by OB.DAAC - amscott » Thu Jan 13, 2022 9:08 am America/New_York

I see a message from the Earthdata helpdesk at the bottom of your script that points to potential issues you should investigate and try to correct. Have you addressed them already?
###
# Issue: ---
# Author: Catalino Cuadrado (catalino.cuadrado@nasa.gov) # Date: 12-19-2018
################################################################################# # Set up R
# You may need to install the httr package.
# install.packages("httr") setwd("C:/Users/ygrund/ygrund_C_Users/PROJECTS/HABs/Branches/2022/ HAB_Shiny_app")
library(httr)
netrc_path <- "./Earthdata/.netrc"
cookie_path <- "./Earthdata/.urs_cookies"
downloaded_file_path <- "./Earthdata/2020/ L2020301.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif"
# Before using the script
#Set up your ~/.netrc file as listed here: https://wiki.earthdata.nasa.gov/ display/EL/How+To+Access+Data+With+cURL+And+Wget set_config(config(followlocation=1,netrc=1,netrc_file=netrc_path,cookie=cookie_path,cookie httr::GET(url = "https://oceandata.sci.gsfc.nasa.gov (/directaccess/CYAN/L3SMI/ 2020/301/L2020301.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif)",
write_disk(downloaded_file_path, overwrite = TRUE))



From that message, you can see that your .netrc file is not where it should be. Your 'home' directory is directly under your user. This path appears to be "C:/Users/ygrund/". Please work through the notes shared by the helpdesk above and try your download again.

ygrund
Posts: 4
Joined: Fri Jan 07, 2022 11:18 am America/New_York
Answers: 0

Re: Download L3SMI data using R

by ygrund » Thu Dec 14, 2023 11:45 pm America/New_York

Hello again, I've returned to the issue of downloading the data. I've noticed that the file directory has been updated. For instance, the link for the file I need to download using R is now: 'https://oceandata.sci.gsfc.nasa.gov/getfile/L2023001.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif'.

I've set up the .netrc and .urs_cookies as per Catalio Cuadrado's instructions (https://wiki.earthdata.nasa.gov/display/EL/How+to+access+data+with+R). The file size should be 136KB, but when using the following code, the downloaded file only has a size of 10KB, indicating an unsuccessful download. Please see my codes below. Any suggestions you could offer would be greatly appreciated.

library(httr)
netrc_path <- "C:/Users/ygrund/ygrund_C_Users/PROJECTS/HABs/HABShiny_Oceandata/.netrc"
cookie_path <- "C:/Users/ygrund/ygrund_C_Users/PROJECTS/HABs/HABShiny_Oceandata/.urs_cookies"
downloaded_file_path <- "C:/Users/ygrund/ygrund_C_Users/PROJECTS/HABs/HABShiny_Oceandata/data/1.tif"

httr::set_config(config(followlocation=1,netrc=1,netrc_file=netrc_path,cookie=cookie_path,cookiefile=cookie_path,cookiejar=cookie_path))

httr::GET(url = "https://oceandata.sci.gsfc.nasa.gov/getfile/L2023001.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_1_1.tif", write_disk(downloaded_file_path, overwrite = TRUE))

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Download L3SMI data using R

by OB.DAAC - SeanBailey » Fri Dec 15, 2023 11:59 am America/New_York

The 10kb file is probably the HTML page for the login redirect. This means something isn't right with the way your credentials are passed, and you were not logged into your EDL account. Verify that your .netrc file is correctly defined.

Sean

ygrund
Posts: 4
Joined: Fri Jan 07, 2022 11:18 am America/New_York
Answers: 0

Re: Download L3SMI data using R

by ygrund » Tue Jan 09, 2024 1:07 pm America/New_York

Hi Sean,

Thank you for your prompt response and for addressing the issue regarding the login redirect. I believe that it might be a problem with the credentials. After several attempts, I have been unable to resolve the issue with the .netrc file while using Catalio Cuadrado's script. I am considering the possibility that it could be related to the IT network configuration within my organization.

However, I have had success in converting the Python script available on the EarthData User website (https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+Python) into an R script, and it works perfectly! :-) In case others encounter a similar problem, I am sharing my script here for reference.

library(httr)

# Function to create a custom session for authentication
session <- function(username, password) {
httr::authenticate(username, password)
}

# Define username and password
username <- "xxxx"
password <- "xxxx"

# Create the session
my_session <- session(username, password)

# URL of the file to retrieve
url <- "https://oceandata.sci.gsfc.nasa.gov/getfile/L2023201.L3m_DAY_CYAN_CI_cyano_CYAN_CONUS_300m_2_2.tif"

# Extract the filename from the URL
filename <- sub(".*/", "", url)

tryCatch({
# Submit the request using the configured session
response <- httr::GET(
url,
httr::progress(),
httr::add_headers(`User-Agent` = "Mozilla/5.0"),
httr::authenticate(username, password)
)

# Check for successful response
if (httr::status_code(response) == 200) {
# Save the file
content <- httr::content(response, as = "raw")
writeBin(content, filename)
print("File downloaded successfully")
} else {
stop("Failed to download the file")
}
}, error = function(e) {
# Handle any errors here
print(paste("Error:", e$message))
})

Thank you again for your assistance.
Yuan

Post Reply