Question about using wget on for Aqua and Merra2

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Mon Feb 06, 2023 10:43 am America/New_York

Hello mgreene,
I attempted the MERRA2 download again, and am still getting the "unable to establish SSL connection" error.
Am I making any syntax errors in the MERRA2 request?
Below is the Windows command line response to my unsuccessful MERRA2 request.

As a sanity check, I also did an AQUA request, which was successful; see Windows command line response for the successful AQUA request also, for reference.

-----------------------------------------------------------------------------------------------------------------------------------------------------
UNSUCCESSFUL MERRA REQUEST:
C:\Users\alcook1\wget>wget --user=anthcook --ask-password --auth-no-challenge=on https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NPASM.5.12.4/2022/01/MERRA2_400.inst3_3d_asm_Np.20220101.nc4
Password for user 'anthcook':
--2023-02-06 10:37:57-- https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NPASM.5.12.4/2022/01/MERRA2_400.inst3_3d_asm_Np.20220101.nc4
Resolving goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)... 198.118.197.96
Connecting to goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)|198.118.197.96|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fgoldsmr5.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYXRhL01FUlJBMi9NMkkzTlBBU00uNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC5pbnN0M18zZF9hc21fTnAuMjAyMjAxMDEubmM0 [following]
--2023-02-06 10:37:57-- https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fgoldsmr5.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYXRhL01FUlJBMi9NMkkzTlBBU00uNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC5pbnN0M18zZF9hc21fTnAuMjAyMjAxMDEubmM0
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
Unable to establish SSL connection.
-----------------------------------------------------------------------------------------------------------------------------------------


-----------------------------------------------------------------------------------------------------------------------------------------
SUCCESSFUL AQUA REQUEST:
C:\Users\alcook1\wget>wget --user=anthcook --ask-password --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc
Password for user 'anthcook':
--2023-02-06 10:36:59-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code [following]
--2023-02-06 10:36:59-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=f871a46667bf06030080ec40607996ff318131c101db6807d0c9c4ab9c6f183e [following]
--2023-02-06 10:37:00-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=f871a46667bf06030080ec40607996ff318131c101db6807d0c9c4ab9c6f183e
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc [following]
--2023-02-06 10:37:00-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 3920979 (3.7M) [application/octet-stream]
Saving to: 'AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc'

AQUA_MODIS.20220101.L3m.DAY.I 100%[=================================================>] 3.74M 6.27MB/s in 0.6s

2023-02-06 10:37:01 (6.27 MB/s) - 'AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc' saved [3920979/3920979]
-----------------------------------------------------------------------------------------------------------------------------------------

Tags:

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Question about using wget on for Aqua and Merra2

by GES DISC - mgreene » Mon Feb 06, 2023 12:03 pm America/New_York

Anthony,
Could you please do wget -V to make sure you are using the newer version and verify that it links to newer SSL packages.

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Mon Feb 06, 2023 12:28 pm America/New_York

mgreene,
The results of wget -V are copied below. Is this a sufficient wget and SSL version?
Anthony

----------------------------------------------------------------------
C:\Users\alcook1\wget>wget -V
GNU Wget 1.21.3 built on mingw32.

+cares +digest +gpgme +https +ipv6 +iri +large-file +metalink -nls
+ntlm +opie +psl +ssl/openssl

Wgetrc:
W:/msys64/home/ender/wget/dep64/etc/wgetrc (system)
Compile:
gcc -DHAVE_CONFIG_H
-DSYSTEM_WGETRC="/home/ender/wget/dep64/etc/wgetrc"
-DLOCALEDIR="/home/ender/wget/dep64/share/locale" -I. -I../../src
-I../lib -I../../lib -I/home/ender/wget/dep64/include
-IW:/msys64/home/ender/wget/dep64/include
-IW:/msys64/home/ender/wget/dep64/include
-IW:/msys64/home/ender/wget/dep64/include -DCARES_STATICLIB
-IW:/msys64/home/ender/wget/dep64/include -DPCRE2_STATIC
-IW:/msys64/home/ender/wget/dep64/include
-IW:/msys64/home/ender/wget/dep64/include -DHAVE_LIBSSL -I
-IW:/msys64/home/ender/wget/dep64/include -DNDEBUG -ggdb
-mtune=broadwell -mtune=znver2 -O2 -pipe -msse -mfpmath=sse
-ffast-math -ftree-vectorize -msse2
Link:
gcc -IW:/msys64/home/ender/wget/dep64/include
-IW:/msys64/home/ender/wget/dep64/include
-IW:/msys64/home/ender/wget/dep64/include -DCARES_STATICLIB
-IW:/msys64/home/ender/wget/dep64/include -DPCRE2_STATIC
-IW:/msys64/home/ender/wget/dep64/include
-IW:/msys64/home/ender/wget/dep64/include -DHAVE_LIBSSL -I
-IW:/msys64/home/ender/wget/dep64/include -DNDEBUG -ggdb
-mtune=broadwell -mtune=znver2 -O2 -pipe -msse -mfpmath=sse
-ffast-math -ftree-vectorize -msse2 -L/home/ender/wget/dep64/lib
-LW:/msys64/home/ender/wget/dep64/lib -lmetalink
-LW:/msys64/home/ender/wget/dep64/lib -lcares
-LW:/msys64/home/ender/wget/dep64/lib -lpcre2-8
-LW:/msys64/home/ender/wget/dep64/lib -lidn2
-LW:/msys64/home/ender/wget/dep64/lib -lssl -lcrypto -L -lz
-LW:/msys64/home/ender/wget/dep64/lib -lpsl -lws2_32 -lole32
-lcrypt32 -lexpat -LW:/msys64/home/ender/wget/dep64/lib -lgpgme
../lib/libgnu.a -lws2_32 -lws2_32 -lws2_32 -lws2_32 -lws2_32
/home/ender/wget/dep64/lib/libiconv.a
/home/ender/wget/dep64/lib/libunistring.a
/home/ender/wget/dep64/lib/libiconv.a -lws2_32

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.

-----------------------------------------------------------------

jmargetts
Posts: 6
Joined: Tue Feb 07, 2023 5:18 am America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by jmargetts » Wed Feb 08, 2023 9:42 am America/New_York

Hello,

I am getting the same error. I'm using the same version of wget is there a preferred version to use?

Regards,

Jordan

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Wed Feb 08, 2023 11:51 am America/New_York

Hello Jordan,
mgreene posted earlier in this thread that "a version of wget 1.18 or greater compiled with gnuTLS 3.3.3 or OpenSSL 1.0.2 or LibreSSL 2.0.2 or later is recommended".

When I ran the command "wget -V", the result was "GNU Wget 1.21.3 built on mingw32", also with a mention of LibSSL in the resulting text.

You might want to run "wget -V" also to see what version you are running.

I have not heard back from mgreene or the support staff yet, so maybe there is still an issue on their end that is being worked through regarding this problem.

Anthony

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Wed Feb 08, 2023 2:08 pm America/New_York

I comm'd with a very helpful MERRA2 staff member offline, and despite many excellent suggestions and attempts, this remains an unresolved issue. The conclusion is that even the newest Windows wget version is unreliable with the SSL protocols of the MERRA2 server (although it works fine with the MODIS-AQUA server).

One recommendation was to methodically download some slightly earlier versions of wget in hopes that one of the slightly earlier versions is not incompatible, or "less" incompatible, with the MERRA2 server protocols.

If I find any combination of wget version and wget command options that ends up working, I will be sure to post a follow up here.

Many thanks for everyone's help,
Anthony

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Question about using wget on for Aqua and Merra2

by GES DISC - mgreene » Wed Feb 08, 2023 2:27 pm America/New_York

Jordan and Anthony, I glad to hear you found my colleague's assistance helpful. Best regards,

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Wed Feb 08, 2023 2:37 pm America/New_York

mgreene,
Yes, and many thanks for all of your comments, time and suggestions.

I will be sure to ping this thread if a solution comes up.
Anthony

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Wed Feb 08, 2023 3:30 pm America/New_York

Good news. The MERRA staff member suggested that I try the Windows wget MERRA2 download with my VNP disconnected, and the download worked like a charm. It seems there was an issue with the MERRA2 server blocking my VPN IP address.

Post Reply