wget issue - "Unable to establish SSL connection"

jdoolin · by **jdoolin** » Tue Feb 14, 2017 10:35 am America/New_York

Hi,

I´ve ordered some data from the website, and recieved the standard email:

Manifest URL:
   https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist103&p=/data1/30c0b0c011ab8656

You can then use wget to download the files in your manifest file:
   wget --content-disposition -i manifest-file-name

You can also pipe wget commands together to initiate the download:
   wget -O - 'manifest-url-from-above' | wget --content-disposition -i -

When I use "wget --content-disposition -i "http_manifest.txt"", it connects to the urls, then returns the error "Unable to establish SSL connection"

I´ve tried using the command --no-check-certificate and all the options for --secure-protocol=protocol on http://www.gnu.org/software/wget/manual/html_node/HTTPS-_0028SSL_002fTLS_0029-Options.html and I can´t get any of them to work.

I´ve also tried
wget -O - 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist103&p=/data1/30c0b0c011ab8656' | wget --content-disposition -i -

And this doesn´t work either, it just returns the message "Unsupported scheme 'p' is not recognised as an internal or external command, oprable programme or batch file"

Please forgive me if I've missed something really obvious, I've never done anything like this before and it's been a very steep learning curve to work out what to do! Any help would be very appreciated

gnwiii · by **gnwiii** » Tue Feb 14, 2017 11:35 am America/New_York

You should post you wget version and platform. The https configuration used by NASA is not supported by older software, so you need a relatively recent version of wget and supporting libraries such as gnutls or openssl. Hopefully someone else is using the same platform and can tell you how to get a working wget configuration.

tintinabula · by **tintinabula** » Tue Feb 14, 2017 12:26 pm America/New_York

The version of wget you have installed in built against GnuTLS which does not support the HTTPS ciphers necessary to connect to the OceanColor HTTPS servers. The following post will give more information.

Wget, Git or Curl built against older versions of GnuTLS can not connect to HTTPS
https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6492

jdoolin · by **jdoolin** » Tue Feb 14, 2017 3:17 pm America/New_York

Hi,

Thanks for the response. That was definitely the cause of the problem. I used this website: http://www.bloggingtips.info/install-wget-windows/ to install the newest version and now it's working!

Thanks so much :)

gnwiii · by **gnwiii** » Wed Feb 15, 2017 7:30 am America/New_York

Glad you got it working. You should read Wget options for NASA https.

oo_processing · by **oo_processing** » Tue Mar 28, 2017 12:28 pm America/New_York

It appears that something has recently changed again with the protocols for getting GOCI and MERIS data.
I am still able to do this by hand. e.g. clicking on the file, and entering my user and passwd info into the authorization page here: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=hNX5KEIeCjXnRUidBTF0Uw&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fecho%2Fgetfile%2Frestrict&response_type=code&required_scope=study_area,country

But our scripts no longer work.

Our scripts use this format (which has worked in the past):
curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --output /tmp/user/M20040441513.L1B_FRS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MER_FRS_1PNUPA20040213_151354_000002732024_00154_10228_8087.N1.bz2

I have tried the exact format found here as well (which is really just the same, but a different order of flags):
https://disc.gsfc.nasa.gov/registration/registration-for-data-access#Curl

curl -n -c ~/.urs_cookies -b ~/.urs_cookies -L --url https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MER_FRS_1PNUPA20040213_151354_000002732024_00154_10228_8087.N1.bz2 -o /tmp/user/M20040441513.L1B_FRS.bz2

In both cases I get a file name M20040441513.L1B_FRS.bz2 returned with the following contents:

<!DOCTYPE html><html lang="en-US"><head><meta http-equiv="content-type" content="text/html;charset=utf-8"><meta name="ROBOTS" content="NOARCHIVE"><title>ERROR @ OceanColor Biology Processing Group (OBPG)</title></head><body link=#323232 vlink=#323232 alink=#323232 style="background-color:#ffffff; color:#323232; font-size:175%"><br><hr color=#323232><center><h1><b>.:. ERROR .:.</b></h1><h2>OceanColor Biology Processing Group (OBPG)</h2><blockquote>Sorry, an error has occurred. Use the back button to return to the previous page or go to the <a href="https://oceancolor.gsfc.nasa.gov">Ocean Color Home Page</a>.</blockquote><br><hr color= #323232></body></html>

A ~/.urs_cookies file has been created after curl containing:

~/.urs_cookies
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

oceandata.sci.gsfc.nasa.gov    FALSE   /    TRUE 0    app-obdaac a2b691bed119b7871c07bd8db3dc707b44885a1b
.earthdata.nasa.gov    TRUE /    FALSE   1490803732 urs_user_already_logged yes
#HttpOnly_urs.earthdata.nasa.gov FALSE   /    FALSE   1490803732 _urs-gui_session N2YvdjVpSjBET1E0L2ozb211aFFjc2ttSDR1OUtCMHNJcjFmMFRpNXlSM0lDcUFTc05yend1QzdEVE5FVU1RMng3ckxNK2J4cUNlS2FFYURaVWZielpwc3JOeXppaVl0Mm5rSmZxWGZuYWVCcHVYck5lZ0kwNVBEYnJ2VWNtd2wxMmplWHBrcDRkTmc5U1JydTE5REdoMmg5QWg2Um03dVR5RVp6OVpsamNrPS0tSVBnY0lqQVZYWHBTUjExMFl5TUJ1UT09--4e344e2d01bf9382c711f28323d9b250d352ba44

curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.21 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

BTW, when I use the wget format this is returned:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MER_FRS_1PNUPA20040213_151354_000002732024_00154_10228_8087.N1.bz2 -O MER_FRS_1PNUPA20040213_151354_000002732024_00154_10228_8087.N1.bz2.wget
--2017-03-28 12:31:23-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MER_FRS_1PNUPA20040213_151354_000002732024_00154_10228_8087.N1.bz2
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /echo/getfile/MER_FRS_1PNUPA20040213_151354_000002732024_00154_10228_8087.N1.bz2 [following]
--2017-03-28 12:31:23-- https://oceandata.sci.gsfc.nasa.gov/echo/getfile/MER_FRS_1PNUPA20040213_151354_000002732024_00154_10228_8087.N1.bz2
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 404 Not Found
2017-03-28 12:31:29 ERROR 404: Not Found.

The ~/.urs_cookies file after the wget command contains:
cat ~/.urs_cookies
# HTTP cookie file.
# Generated by Wget on 2017-03-28 12:31:29.
# Edit at your own risk.

.earthdata.nasa.gov    TRUE /    FALSE   1490803732 urs_user_already_logged yes
oceandata.sci.gsfc.nasa.gov    FALSE   /    TRUE 0    app-obdaac a2b691bed119b7871c07bd8db3dc707b44885a1b

Please advise,
Thanks, Brock

by **OB ODPS - towens** » Tue Mar 28, 2017 3:33 pm America/New_York

I have checked two of the files referenced in your post and neither one appears on our archive.

MER_FRS_1PNUPA20040213_151354_000002732024_00154_10228_8087.N1.bz2
M20040441513.L1B_FRS.bz2

Are you working with an outdated list of files?

Tommy

OB.DAACx - amscott · by **OB.DAACx - amscott** » Tue Mar 28, 2017 3:38 pm America/New_York

Hi Brock,

Using our File Search Utility, you can enter in relevant parts of the filename you requested along with the wildcard character (MER_FRS_1P*20040213_151354_000002732024*) and find one granule matching that date/time.

File found: MER_FRS_1PPEPA20040213_151354_000002732024_00154_10228_9587.N1.bz2

Please check to see if this is the actual file you are trying to obtain.

Alicia

oo_processing · by **oo_processing** » Tue Mar 28, 2017 4:05 pm America/New_York

Alicia et al.

Yes, the file was from an old list. I was passed the problem to post thus I didn't check that. I assumed too much :wink:

The actual problem was for a GOCI file, but I asked for a test case, and was given this MERIS file.

I just tried the GOCI file I found here:
https://oceancolor.gsfc.nasa.gov/cgi/browse.pl?sub=level1or2&per=DAY&day=15386&prm=CHL&set=10&mon=15371&sen=g1&rad=0&frc=0&n=37.6171875&w=129.0234375&id=1490731022.11555&cn=38.24&cs=36.31&cw=127.27&ce=129.21&dnm=D&file=G2012047051641.L2_COMS_OC_CHLOR_A_BRS&ndx=2&wid=796&hgt=813

and put it in the curl command:

curl -n -c ~/.urs_cookies -b ~/.urs_cookies -L --url https://oceandata.sci.gsfc.nasa.gov/goci/getfile/COMS_GOCI_L1B_GA_20120216051641.he5.gz -o /tmp/test/COMS_GOCI_L1B_GA_20120216051641.he5.gz

And it worked fine.

I would like to use the file search utility, but GOCI is not a listed mission. Will the all missions find GOCI files?

I guess it does :grin:
Thanks,
Brock