Search and download data files
Posted: Fri Oct 15, 2021 1:59 pm America/New_York
Dear Lady/Sir
I had left several months ago some software running around the clock to work with oceancolor binned data files from Aqua, Terra and SNPP satellites at the Portuguese Institute for the Ocean and Atmosphere, Lisbon, Portugal.
Recently, a colleague asked me to find out why the control script could not find and download data files as before. This is then the motivation of my message addressed now to the Forum.
My script has worked fine with curl (for a long time) and two local files, .netrc and .urs_cookies as explained in your Search and Download instructions available in Direct Data section of the OBPG website.
Inside the curl command I first call your function search_file to check if the given filename exists. If it exists then I issue a second curl command. In this curl command I use your function get_file for downloading it. In case the given filename does not exists (please note that your services have modified the names of many oceancolor data files about 2 years ago) then I take the alternative name and repeat the curl command to activate your function search_file. In case of not found, I still try the same last name for the Near Real Time version.
This has worked quite well for many months until about last summer or so (I believe when the OBPG website was integrated in the Earthdata website).
Please take a look to my curl commands below, where I insert calls to your own function search_file (and get_file). In my script, after calling the curl command for searching the given filename, the connection aborts 60 seconds after CONNECT. The curl issues the error message
curl: (56) Received HTTP code 503 from proxy after CONNECT
That error message is issued after the curl command for searching the file
result1=$(curl -d "sensor=${SENSOR}&sdate=${YYYY1}-${MM1}-${DD1}&edate=${YYYY2}-${MM2}-${DD2}&dtype=L3b&addurl=1&results_as_file=1&search=${old_name}" https://oceandata.sci.gsfc.nasa.gov/api/file_search | head -1)
My questions are as follows, please:
(1) Is NASA server name correct ?
(2) May I still work with your "file_search" executable as before, or some permission is required ?
(3) In case the executable "file_search" is no longer available, could you please tell me what should I do to checking if a given filename exists before trying to download the respective file, please ?
(4) May I still work with your function "get_file" to download a file using the curl command like the following?
$HOME/local/bin/curl -b $HOME/.urs_cookies -c $HOME/.urs_cookies -L -n -O https://oceandata.sci.gsfc.nasa.gov/cgi ... /$old_name
Please note the locations api of file_search, and cgi of get_file functions. Are they still correct?
I thank you for the kindness of your attention and help.
Kind regards and good weekend.
Arlindo Arriaga
PhD Meteorology UW-Madison 1991
I had left several months ago some software running around the clock to work with oceancolor binned data files from Aqua, Terra and SNPP satellites at the Portuguese Institute for the Ocean and Atmosphere, Lisbon, Portugal.
Recently, a colleague asked me to find out why the control script could not find and download data files as before. This is then the motivation of my message addressed now to the Forum.
My script has worked fine with curl (for a long time) and two local files, .netrc and .urs_cookies as explained in your Search and Download instructions available in Direct Data section of the OBPG website.
Inside the curl command I first call your function search_file to check if the given filename exists. If it exists then I issue a second curl command. In this curl command I use your function get_file for downloading it. In case the given filename does not exists (please note that your services have modified the names of many oceancolor data files about 2 years ago) then I take the alternative name and repeat the curl command to activate your function search_file. In case of not found, I still try the same last name for the Near Real Time version.
This has worked quite well for many months until about last summer or so (I believe when the OBPG website was integrated in the Earthdata website).
Please take a look to my curl commands below, where I insert calls to your own function search_file (and get_file). In my script, after calling the curl command for searching the given filename, the connection aborts 60 seconds after CONNECT. The curl issues the error message
curl: (56) Received HTTP code 503 from proxy after CONNECT
That error message is issued after the curl command for searching the file
result1=$(curl -d "sensor=${SENSOR}&sdate=${YYYY1}-${MM1}-${DD1}&edate=${YYYY2}-${MM2}-${DD2}&dtype=L3b&addurl=1&results_as_file=1&search=${old_name}" https://oceandata.sci.gsfc.nasa.gov/api/file_search | head -1)
My questions are as follows, please:
(1) Is NASA server name correct ?
(2) May I still work with your "file_search" executable as before, or some permission is required ?
(3) In case the executable "file_search" is no longer available, could you please tell me what should I do to checking if a given filename exists before trying to download the respective file, please ?
(4) May I still work with your function "get_file" to download a file using the curl command like the following?
$HOME/local/bin/curl -b $HOME/.urs_cookies -c $HOME/.urs_cookies -L -n -O https://oceandata.sci.gsfc.nasa.gov/cgi ... /$old_name
Please note the locations api of file_search, and cgi of get_file functions. Are they still correct?
I thank you for the kindness of your attention and help.
Kind regards and good weekend.
Arlindo Arriaga
PhD Meteorology UW-Madison 1991