"Unable to locally verify the issuer's authority" when using wget

[fcaonyy]

Hi, I was trying to download daily L2 data for one month from OBPG website. In the terminal window, I typed
wget -O - 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist105&p=/data1/8040709090b8815c'| wget --content-disposition -i -

and it returned:

--2017-03-03 15:32:45--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist105&p=/data1/8040709090b8815c
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
ERROR: cannot verify oceandata.sci.gsfc.nasa.gov's certificate, issued by ‘CN=COMODO ECC Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB’:
  Unable to locally verify the issuer's authority.
To connect to oceandata.sci.gsfc.nasa.gov insecurely, use `--no-check-certificate'.
No URLs found in -.

So what should I do to make this work?
I upgraded my system to macOS Sierra version 10.12.3 and I also upgraded the wget stuff most to date.
Thanks for looking into this issue.

[OB.DAAC - SeanBailey]

Which version of wget?  Using the following (installed via macports) works for me (under 10.12.3):

###########################################
$ which wget
/opt/local/bin/wget
gs616-a-prospero3:junk swbaile1$ wget --version
GNU Wget 1.19.1 built on darwin16.4.0.

-cares +digest -gpgme +https +ipv6 -iri +large-file -metalink +nls
+ntlm +opie -psl +ssl/gnutls
....
###########################################

Sean

[OB.DAAC - SeanBailey]

Alternatively, this works for me as well:

for file in $(/usr/bin/curl 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist105&p=/data1/8040709090b8815c' );
do
  echo $file
  /usr/bin/curl -L -J -O "$file";
done;

Sean

[fcaonyy]

Thanks for looking into this, Sean. I checked the wget, and the version is

GNU Wget 1.18 built on darwin15.5.0.

-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
+ntlm +opie -psl +ssl/openssl

I tried to updated to 1.19 as yours, but after installation, it still shows the above to me. Am I doing something wrong?
I tried to acquire data from the web again this morning, and the error I got was the same as last Friday

ERROR: cannot verify oceandata.sci.gsfc.nasa.gov's certificate, issued by ‘CN=COMODO ECC Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB’:
  Unable to locally verify the issuer's authority.
To connect to oceandata.sci.gsfc.nasa.gov insecurely, use `--no-check-certificate'.
No URLs found in -.

So another question is: how to do "--no-check-certificate" in terminal ?
Thanks again.
  Fang

[gnwiii]

For https URL's, wget needs an SSL/TLS library, usually openssl, and certificates.  I use Macports's wget on darwin 15.6.0, linked to Macports' openssl.  The certifcates are in /opt/local/etc/openssl and come from the curl-ca-bundle package:

Description:          Installs a bundle of certification authority certificates
                      (CA certs) which curl (when linked with OpenSSL) uses to
                      verify the authenticity of secure web and FTP servers.
Homepage:             https://curl.haxx.se


See: Stackoverflow -- how do I fix certificate errors...? started with a Cygwin question but has some MacOS pointers.

[fcaonyy]

Hi, gnwiii, thanks for your reply. I tried to install certificates following another web since this one is more straightforward:

https://support.securly.com/hc/en-us/articles/206058318-Securly-SSL-certificate-manual-install-in-Mac-OS-X

After installation, I typed in the terminal

wget -O - 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data2/60709080d3b18a54'| wget --content-disposition -i -
and I got the same error again:

--2017-03-06 17:47:02--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data2/60709080d3b18a54
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
ERROR: cannot verify oceandata.sci.gsfc.nasa.gov's certificate, issued by ‘CN=COMODO ECC Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB’:
  Unable to locally verify the issuer's authority.
To connect to oceandata.sci.gsfc.nasa.gov insecurely, use `--no-check-certificate'.
No URLs found in -.

So am I talking about the same certificate as yours ? Thanks again.
  Fang

[fcaonyy]

Also, if I want to do it in a quick way, where should I insert "--no-check-certificate" in the wget command line?
Thanks.

[gnwiii]

The "quick fix" get around an incomplete or broken installation is to remove the broken software and install a "known working" configuration (Anaconda python 2.7, which includes curl or macports' python 2.7 with git, wget, and curl).   The "--no-check-certificates" option to wget defeats the purpose of using https and won't make the processing system work as https URL's are also used by python scripts and git.

[fcaonyy]

I still cannot make it work. I searched in the forum and found this post: https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6423
I followed what monger said and did this in the terminal
wget --no-check-certificate --load-cookies ~/.urs_cookies --save-cookies  ~/.urs_cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S19980011998031.L3m_MO_CHL_chl_ocx_9km.nc

It worked and the file was downloaded successfully.

However, for my own file I did the same thing, just replaced the final link as

wget --no-check-certificate --load-cookies ~/.urs_cookies --save-cookies  ~/.urs_cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist102&p=/data1/7000c0d0e5b18051

it shows error like:

WARNING: cannot verify oceandata.sci.gsfc.nasa.gov's certificate, issued by ‘CN=COMODO ECC Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 409 Conflict
2017-03-07 11:42:55 ERROR 409: Conflict.

Any comments to solve this problem? Thanks.

[OB.DAAC - SeanBailey]

First, unless you are downloading L1B data from either MERIS, GOCI, HICO or OLCI , you do not need to set up the URS cookies for wget....doesn't hurt, but isn't required for most products.

Since the --no-check-certificate worked for that call, it should work for you in general.  The 409 conflict error is saying you've got a problem with the request.  That problem is that you didn't encapsulate the URL in quotes, so the shell intercepted and attempted to interpret the '&'.

Try:
wget --no-check-certificate  "https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist102&p=/data1/7000c0d0e5b18051"

As an aside, gnwiii has a point...using --no-check-certificate is not ideal and you probably want to figure out why the wget installation is having an issue with the certificate...I can assure you it is valid.

Sean