Wget, Git or Curl built against older versions of GnuTLS can not connect to HTTPS

[tintinabula]

Older versions of GnuTLS do not support modern, secure ciphers used by the OceanColor (OBPG) HTTPS servers. If you are using a client like Wget, Git or Curl built against an older version of GnuTLS then you may see an error similar to, "Unable to establish SSL connection" or "cipher not supported". The OceanColor HTTPS servers require AES 128 bit or AES 256 bit  Galois/Counter Mode (GCM) ciphers.

To query if your client is built against gnutls or openssl use the "--verbose" argument with the client binary. For example, wget on Ubuntu 16.04 LTS is built against OpenSSL.

$ wget -V
GNU Wget 1.17.1 built on linux-gnu.
Link:
  .... openssl.o

We recommend updating your operating system (OS) to receive the latest versions of the client binaries and GnuTLS libraries as well as any security updates released by the developers. Updating to the latest patches may give you newer binaries which support newer ciphers. Another option is upgrading to the latest version of your OS. For example, if you are using Ubuntu 12.04 then upgrade to Ubuntu 16.04 which supports the newest ciphers and is able to connect to the OceanColor servers without issue. Check with your local system administrator if you need help.

If you can not update your OS then we recommend building your download client against OpenSSL or the newest version of GnuTLS  downloaded from the GnuTLS website. Search Google for "build wget from source" or "build git from source" for many public tutorials.

[mkahru]

Hi,
My Wget script on Windows has worked for years but no more. For each input file in http_manifest.txt it produces a text file with a very long name like authorize@response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa
I downloaded the latest Wget for 64bit Windows but it did not help. Any ideas? My script is the following:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --content-disposition -i http_manifest.txt
As i said, it used to work fine in the past.
Thanks !
I checked the wget version and it is the latest and also has Link:
-lgpg-error ftp-opie.o mswindows.o openssl.o http-ntlm.o
The full output is:
GNU Wget 1.20.3 built on mingw32.

-cares +digest +gpgme +https +ipv6 +iri +large-file +metalink -nls
+ntlm +opie +psl +ssl/openssl

Wgetrc:
    /win32dev/misc/wget/out64/etc/wgetrc (system)
Compile:
    x86_64-w64-mingw32-gcc -DHAVE_CONFIG_H
    -DSYSTEM_WGETRC="/win32dev/misc/wget/out64/etc/wgetrc"
    -DLOCALEDIR="/win32dev/misc/wget/out64/share/locale" -I. -I../lib
    -I../lib -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
Link:
    x86_64-w64-mingw32-gcc -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
    -L/win32dev/misc/wget/out64/lib -L/win32dev/misc/wget/out64/lib
    -lmetalink -lunistring -liconv -L/win32dev/misc/wget/out64/lib
    -lpcre2-8 -lidn2 -L/win32dev/misc/wget/out64/lib -lssl -lcrypto
    -L/win32dev/misc/wget/out64/lib -lz -L/win32dev/misc/wget/out64/lib
    -lpsl -lws2_32 -lunistring -lws2_32 -lole32 -lcrypt32 -lexpat
    -L/win32dev/misc/wget/out64/lib -lgpgme -lassuan -lws2_32
    -lgpg-error ftp-opie.o mswindows.o openssl.o http-ntlm.o
    ../lib/libgnu.a -lws2_32 -lws2_32 -lws2_32 -lws2_32 -lws2_32
    /win32dev/misc/wget/out64/lib/libiconv.a
    /win32dev/misc/wget/out64/lib/libunistring.a -lws2_32

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.

[monger]

Hi--- I think I have my secure file transfer credentials all set correctly and I can download the example file (single file method) using wget and also using curl that you give in the methods page for secure file transfer.  However, I am having trouble retrieving extracted files that I ordered through the L1/L2 data ordering system using the same secure wget or curl approach that works with the Direct Access Data files given in your example pages.  Could you provide an example of secure wget and/or secure curl command that will work with the files below?  Thanks!!!!  :)   One more thing, I can download the files below via Safari.  And I am using wget version 1.20.3

https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2012240172500.L2_LAC_OC.x.nc?h=ocdist205&p=/data1/d090d0a04422e150/requested_files
https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2012243175500.L2_LAC_OC.x.nc?h=ocdist205&p=/data1/d090d0a04422e150/requested_files

Bruce

[gnwiii]

There are many versions of wget for Windows.   The Earthdata WIKI has  Eternallybored.org where there is wget version 1.20.3 OpenSSL 1.1.1b, ZLib 1.2.11, gpgme-1.13.0, pcre2 10.32, libpsl 0.20.2, taskbar progressbar, Windows certificate store support, manual; the binaries have been updated and now work fine with SSL connections.   This version behaves as you describe.

The above Earthdata WIKI page also suggests using Cygwin on Windows.  Cygwin GNU Wget 1.19.1 works for me:
$ wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition "https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2020013.1200.003"
--2020-01-21 09:44:00--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2020013.1200.003
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/PM1EPHND.P2020013.1200.003 [following]
--2020-01-21 09:44:00--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/PM1EPHND.P2020013.1200.003
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&required_scope=study_area,country [following]
--2020-01-21 09:44:01--  https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=c98a113252b194e4a3fdb09ee25d0f6879ab05b685f131af5a4052f1d77ea92c [following]
--2020-01-21 09:44:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=c98a113252b194e4a3fdb09ee25d0f6879ab05b685f131af5a4052f1d77ea92c
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/PM1EPHND.P2020013.1200.003 [following]
--2020-01-21 09:44:02--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/PM1EPHND.P2020013.1200.003
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 5531136 (5.3M) [application/octet-stream]
Saving to: ‘PM1EPHND.P2020013.1200.003’

PM1EPHND.P2020013.1200.003    100%[=================================================>]   5.27M  1.13MB/s    in 4.9s

2020-01-21 09:44:12 (1.09 MB/s) - ‘PM1EPHND.P2020013.1200.003’ saved [5531136/5531136]


[emaure]

The authentication has brought a lot of issues and I struggled for a day :cry:.

After many trials with the examples provided https://oceancolor.gsfc.nasa.gov/data/download_methods/, I was able to download on windows machine using session cookies

It seems that wget fails without the "--auth-no-challenge=on" flag set. In addition, the approach to use cookies fails even using the Windows Subsystem Linux (WSL) which offers a full linux command line.

Here, I am just sharing my success story on Windows 10 build 1809 64-Bit with wget 1.20.3 (https://eternallybored.org/misc/wget/)

Instead of using the command

> echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc ; > ~/.urs_cookies
> chmod  0600 ~/.netrc

Create these files in your ~/
Replace the '~/' with windows C:\Users\username\

> 1. wget --user=user--password=password --keep-session-cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on url
> 2. wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition url

Skipping step (1), (2) fails even if you have run the echo command.
Everytime I executed "echo" and (2) I failed. But executing (1) and (2) I was successful.
Note that (1) only needs to be executed once, just like "echo"

[schmidtga]

I recently had similar issues to those commenting above; my download scripts using wget worked just fine but as of Jan. 7 the downloads were text files instead of the HDF contents.  I had to add "--auth-no-challenge=on --content-disposition" to my wget command in order to get my scripts to work again.

Example:
wget --user=user_name --password=my_password --no-verbose --auth-no-challenge=on --content-disposition https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/N202000800_MET_NCEP_6h.hdf

[sararivero]

i am having the same problem, did you figure it out?
Thanks!

[OB.DAACx - SeanBailey]

Have you seen: https://oceancolor.gsfc.nasa.gov/data/download_methods/ ?

[sararivero]

Hi,

Yes, Sean. I am having a similar problem as "mkahru"....
I received an email saying that my data is staged. I downloaded the manifest file using the following link:

   https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist204&p=/data1/e090d07005a4ef57

then i followed the email instructions to download files using the manifest: 

wget --content-disposition -i new_manifest.txt

and like this:  

wget -O - ' https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist204&p=/data1/e090d07005a4ef57' | wget --content-disposition -i -

Neither worked. I have tried using " ", ' '. I also tried:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --content-disposition -i new_manifest.txt

and it downloads a weird thing "authorize?response_type=code&redirect_uri=https/%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA.

I tried adding user and password  and same result:

wget --user=user--password=password --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --content-disposition -i new_manifest.txt

I have looked for other peoples suggestions in the forum but none of them work. With this other trial below it says "Redirecting output to ‘wget-log’." and nothing else happens:
wget --user=username--password=password --keep-session-cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist204&p=/data1/e090d07005a4ef57

What am I doing wrong? My authentication is not the problem as I am able to download individual images.

Thank you in advance

[OB.DAACx - SeanBailey]

Have you defined a .netrc file with your EarthData Login credentials?  In your last example, you probably need to quote the URL, as the "&" may confuse the shell into thinking you are stringing two commands together.
Also, wget can be cranky.  Different versions can behave differently.  There is a python script posted on the download_methods page that is much more robust.  You may want to try it.

Sean