Question about using wget on for Aqua and Merra2

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Question about using wget on for Aqua and Merra2

by anthcook » Thu Feb 02, 2023 5:36 pm America/New_York

Hello all, I am new at using wget for downloading data from a Windows command prompt using wget.

I was able to generate a user/password cookie file using either of the the two commands:
wget -qO- --keep-session-cookies --save-cookies cookies.txt --post-data --user=USERNAME --ask-password https://urs.earthdata.nasa.gov/
OR
wget -qO- --keep-session-cookies --save-cookies cookies.txt --post-data --user=USERNAME password=PASSWORD https://urs.earthdata.nasa.gov/


For downloading Aqua data with wget, any of the commands below worked well:
wget URL
wget --user=USERNAME --ask-password URL
wget --user=USERNAME --ask-password --auth-no-challenge=on URL
(where URL = https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc).

As can be seen above, downloading the Aqua file did not care whether or not I used the call to the user/password, but it also worked fine with the call to the user/password.



However, when trying to download Merra2 data, I keep getting the error "Unable to establish SSL connection", regardless of how I configure the wget line:
wget URL
wget --user=USERNAME --ask-password URL
wget --user=USERNAME --ask-password --auth-no-challenge=on URL
(where URL = https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NPASM.5.12.4/2022/01/MERRA2_400.inst3_3d_asm_Np.20220101.nc4)

My approved applications on the EarthData site are:
OB.DAAC Modis
Earthdata Feedback Module
Earthdata Code Collaborative
Earthdata Website
OB.DAAC User Support Services
GESDISC Test Data Archive
OB.DAAC L3 Browser
CMR SSO APP for EDL in PROD
Metadata Management Tool
NASA GESDISC DATA ARCHIVE
Earthdata Forum


What am I doing wrong with trying to download Merra2 data with wget?

Tags:

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Question about using wget on for Aqua and Merra2

by GES DISC - mgreene » Fri Feb 03, 2023 8:24 am America/New_York

Hello, in regards to downloading MERRA data using wget, a version of wget 1.18 or greater compiled with gnuTLS 3.3.3 or OpenSSL 1.0.2 or LibreSSL 2.0.2 or later is recommended. I'm also including data access instructions as follows, https://disc.gsfc.nasa.gov/data-access

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Fri Feb 03, 2023 10:05 am America/New_York

Thank you, I will check my wget version and investigate the other versions you mention if that seems to be the issue.

Back to the original post, can someone also help me understand why I can download the MODIS-Aqua file either with or without the password functions embedded in the wget line? Are passwords simply not required for this particular data set? Or are there other higher-level dynamics at work?

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Fri Feb 03, 2023 11:13 am America/New_York

I am fairly certain that I was running a version of wget that is later that ver 1.18, but to be sure, I downloaded ver 1.21.3, 64bit.
If I am reading the download information correctly (see attachment), ver1.21.3 was compiled with OpenSSL 1.1.1m.

Upon the attempt to download MERRA2 data with wget ver1.21.3, I was able to successfully download the MERRA2 file.

However, every attempt after that failed. Any insights as to whether the failure to connect is happening on my end (wget) or the MERRA2 server end?


Here is the Windows command-line code returned after the successful and MERRA2 downloads:

SUCCESSFUL MERRA2 DOWNLOAD:
C:\Users\alcook1\wget>wget --user=anthcook --ask-password --auth-no-challenge=on https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NPASM.5.12.4/2022/01/MERRA2_400.inst3_3d_asm_Np.20220101.nc4
Password for user 'anthcook':
--2023-02-03 10:42:18-- https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NPASM.5.12.4/2022/01/MERRA2_400.inst3_3d_asm_Np.20220101.nc4
Resolving goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)... 198.118.197.96
Connecting to goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)|198.118.197.96|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fgoldsmr5.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYXRhL01FUlJBMi9NMkkzTlBBU00uNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC5pbnN0M18zZF9hc21fTnAuMjAyMjAxMDEubmM0 [following]
--2023-02-03 10:42:18-- https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fgoldsmr5.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYXRhL01FUlJBMi9NMkkzTlBBU00uNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC5pbnN0M18zZF9hc21fTnAuMjAyMjAxMDEubmM0
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://goldsmr5.gesdisc.eosdis.nasa.gov/data-redirect?code=2ec815a5430118ebf3f6402497d483ed1ddd611256e1b225b3d3ac200c076c78&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYXRhL01FUlJBMi9NMkkzTlBBU00uNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC5pbnN0M18zZF9hc21fTnAuMjAyMjAxMDEubmM0 [following]
URL transformed to HTTPS due to an HSTS policy
--2023-02-03 10:42:19-- https://goldsmr5.gesdisc.eosdis.nasa.gov/data-redirect?code=2ec815a5430118ebf3f6402497d483ed1ddd611256e1b225b3d3ac200c076c78&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYXRhL01FUlJBMi9NMkkzTlBBU00uNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC5pbnN0M18zZF9hc21fTnAuMjAyMjAxMDEubmM0
Connecting to goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)|198.118.197.96|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NPASM.5.12.4/2022/01/MERRA2_400.inst3_3d_asm_Np.20220101.nc4 [following]
--2023-02-03 10:42:20-- https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NPASM.5.12.4/2022/01/MERRA2_400.inst3_3d_asm_Np.20220101.nc4
Connecting to goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)|198.118.197.96|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1208939565 (1.1G)
Saving to: 'MERRA2_400.inst3_3d_asm_Np.20220101.nc4'
MERRA2_400.inst3_3d_asm_Np.20 100%[=================================================>] 1.12G 8.46MB/s in 2m 55s
2023-02-03 10:45:15 (6.58 MB/s) - 'MERRA2_400.inst3_3d_asm_Np.20220101.nc4' saved [1208939565/1208939565]


UNSUCCESSFUL MERRA2 DOWNLOAD:
C:\Users\alcook1\wget>wget --user=anthcook --ask-password --auth-no-challenge=on https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NPASM.5.12.4/2022/01/MERRA2_400.inst3_3d_asm_Np.20220101.nc4
Password for user 'anthcook':
--2023-02-03 10:54:36-- https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2/M2I3NPASM.5.12.4/2022/01/MERRA2_400.inst3_3d_asm_Np.20220101.nc4
Resolving goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)... 198.118.197.96
Connecting to goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)|198.118.197.96|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fgoldsmr5.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYXRhL01FUlJBMi9NMkkzTlBBU00uNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC5pbnN0M18zZF9hc21fTnAuMjAyMjAxMDEubmM0 [following]
--2023-02-03 10:54:36-- https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fgoldsmr5.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYXRhL01FUlJBMi9NMkkzTlBBU00uNS4xMi40LzIwMjIvMDEvTUVSUkEyXzQwMC5pbnN0M18zZF9hc21fTnAuMjAyMjAxMDEubmM0
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
Unable to establish SSL connection.
Attachments
Wget version info.PNG
Wget version info.PNG (38.98 KiB) Not viewed yet

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Question about using wget on for Aqua and Merra2

by OB.DAAC - SeanBailey » Fri Feb 03, 2023 12:07 pm America/New_York

To answer the question about needing (or not) to pass username/password to wget for the SST data download...you do need to do so. Did you look at the file you downloaded without the credentials? What I'm pretty certain you will find is that it is NOT a netCDF file, but rather an HTML file that is the login page for Earthdata.

Sean

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Fri Feb 03, 2023 12:35 pm America/New_York

Sean,
Great catch, thank you.
Whether or not I use the password credentials in the wget command, the result is a file with the .nc extension:
AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc

However, the size of the .nc file is only 13kB without the pwd credentials, and is 10MB with the credentials.
Since the no-credential result was a file with a .nc extension (albeit smaller in size), I was not noticing that the size was much smaller than the size when the pwd credentials were applied.

The 10MB .nc file could be opened and inspected with HDFView, but the 13kB .nc file produced an error when attempting to open with HDFView. The 13kB file with the .nc extension was indeed an html file that could be opened with an HTML app.

See below for the returned command line code for the wget command with and without the password credentials:
Anthony

WITHOUT PWD CREDENTIALS:
C:\Users\alcook1\wget>wget https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc
--2023-02-03 12:20:04-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code [following]
--2023-02-03 12:20:04-- https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc'

AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc [ <=> ] 12.12K --.-KB/s in 0s
2023-02-03 12:20:05 (46.8 MB/s) - 'AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc' saved [12412]



WITH PWD CREDENTIALS:
C:\Users\alcook1\wget>wget --user=anthcook --ask-password --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc
Password for user 'anthcook':
--2023-02-03 12:22:03-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict [following]
--2023-02-03 12:22:03-- https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=367932b73761f61dde0e4766cd1dc4fcb3fa307bf46b8a352ef51449cec73bfc [following]
--2023-02-03 12:22:04-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=367932b73761f61dde0e4766cd1dc4fcb3fa307bf46b8a352ef51449cec73bfc
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc [following]
--2023-02-03 12:22:04-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 10725977 (10M) [application/octet-stream]
Saving to: 'AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc'

AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc 100%[============================================================================================================================>] 10.23M 9.20MB/s in 1.1s

2023-02-03 12:22:06 (9.20 MB/s) - 'AQUA_MODIS.20211202.L3m.DAY.SST4.sst4.4km.nc' saved [10725977/10725977]

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Question about using wget on for Aqua and Merra2

by OB.DAAC - SeanBailey » Fri Feb 03, 2023 5:03 pm America/New_York

Whether or not I use the password credentials in the wget command, the result is a file with the .nc extension:
Yes. The wget program has no idea what filetype is being sent, just that you requested some file (with a .nc extension) and some data was sent to you (so it saved it to a file with the name of what it expected was being sent).

We'll look to see if there is a way for the query on our end to bail without sending the login redirect page if the request comes from a non-interactive utility...but that might be tough. You see, the web server doesn't care about the client connecting...it just knows to request credentials if not provided, so it does so.

Sean

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Question about using wget on for Aqua and Merra2

by anthcook » Fri Feb 03, 2023 6:10 pm America/New_York

Sean,
Thank you; makes sense. Now that I am aware, and at least from my perspective, there is no need for any further tweaking on your end.

I'm also still wondering why I had one successful MERRA2 download with wget, but for all other (many attempts), it fails to establish an SSL connection?

Are there any hints that can be obtained from the text returned (see previous post above) in the Windows command window for the SUCCESSFUL MERRA2 DOWNLOAD compared to the UNSUCCESSFUL MERRA2 DOWNLOAD?

Anthony

GES DISC - jimacker
User Services
User Services
Posts: 298
Joined: Mon Sep 30, 2019 1:57 pm America/New_York
Answers: 3
Has thanked: 2 times
Been thanked: 4 times

Re: Question about using wget on for Aqua and Merra2

by GES DISC - jimacker » Fri Feb 03, 2023 8:04 pm America/New_York

Our staff is examining the information you provided about the MERRA-2 download and will reply, likely early next week (February 6-10).

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Question about using wget on for Aqua and Merra2

by GES DISC - mgreene » Mon Feb 06, 2023 8:59 am America/New_York

Anthony,
Please try your MERRA data request again, if you still receive an error check to see if you can download the file from the Web Browser to verify that Earthdata Login credentials work. thanks

Post Reply