wget Unable to establish SSL connection for MERRA2 data

nwiseman · by **nwiseman** » Fri Mar 03, 2023 5:50 pm America/New_York

Hi,
I am attempting to download subset MERRA2 data for a series of files (n = 214) and have been having issues with downloading subset data due to the destination name being too long. I am running on a windows machine with wget v1.21.31 (in my Windows/System32 directory). My input is the following:
> wget --load-cookies C:\Users\niciw\OneDrive\Documents\MATLAB\ocean_color\data\MERRA2\.urs_cookies --save-cookies C:\Users\niciw\OneDrive\Documents\MATLAB\ocean_color\data\MERRA2\.urs_cookies --auth-no-challenge=on --keep-session-cookies --user=nwiseman --ask-password --content-disposition -i MERRA2_Atlantic.txt
where MERRA2_Atlantic.txt is my list of files from GES DISC Data Access.
For example, one file name is: https://goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2_DIURNAL/M2TUNXADG.5.12.4/2006/MERRA2_300.tavgU_2d_adg_Nx.200602.nc4.nc4?BCDP001[0:23][180:250][192:272],BCDP002[0:23][180:250][192:272],BCWT001[0:23][180:250][192:272],BCWT002[0:23][180:250][192:272],DUDP001[0:23][180:250][192:272],DUDP002[0:23][180:250][192:272],DUDP003[0:23][180:250][192:272],DUDP004[0:23][180:250][192:272],DUDP005[0:23][180:250][192:272],DUWT001[0:23][180:250][192:272],DUWT002[0:23][180:250][192:272],DUWT003[0:23][180:250][192:272],DUWT004[0:23][180:250][192:272],DUWT005[0:23][180:250][192:272],time,lat[180:250],lon[192:272]
When I run the command, the initial file (the README.pdf) downloads with no issues, and then as soon as it gets to the next file I get the following:
https://goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2_DIURNAL/M2TUNXADG.5.12.4/2006/MERRA2_300.tavgU_2d_adg_Nx.200602.nc4.nc4?BCDP001[0:23][180:250][192:272],BCDP002[0:23][180:250][192:272],BCWT001[0:23][180:250][192:272],BCWT002[0:23][180:250][192:272],DUDP001[0:23][180:250][192:272],DUDP002[0:23][180:250][192:272],DUDP003[0:23][180:250][192:272],DUDP004[0:23][180:250][192:272],DUDP005[0:23][180:250][192:272],DUWT001[0:23][180:250][192:272],DUWT002[0:23][180:250][192:272],DUWT003[0:23][180:250][192:272],DUWT004[0:23][180:250][192:272],DUWT005[0:23][180:250][192:272],time,lat[180:250],lon[192:272]
Connecting to goldsmr4.gesdisc.eosdis.nasa.gov (goldsmr4.gesdisc.eosdis.nasa.gov)|198.118.197.95|:443... connected.
HTTP request sent, awaiting response... 302 Found
The destination name is too long (608), reducing to 241
Location: https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fgoldsmr4.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNC5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9vcGVuZGFwL01FUlJBMl9ESVVSTkFML00yVFVOWEFERy41LjEyLjQvMjAwNi9NRVJSQTJfMzAwLnRhdmdVXzJkX2FkZ19OeC4yMDA2MDIubmM0Lm5jND9CQ0RQMDAxWzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLEJDRFAwMDJbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sQkNXVDAwMVswOjIzXVsxODA6MjUwXVsxOTI6MjcyXSxCQ1dUMDAyWzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLERVRFAwMDFbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sRFVEUDAwMlswOjIzXVsxODA6MjUwXVsxOTI6MjcyXSxEVURQMDAzWzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLERVRFAwMDRbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sRFVEUDAwNVswOjIzXVsxODA6MjUwXVsxOTI6MjcyXSxEVVdUMDAxWzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLERVV1QwMDJbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sRFVXVDAwM1swOjIzXVsxODA6MjUwXVsxOTI6MjcyXSxEVVdUMDA0WzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLERVV1QwMDVbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sdGltZSxsYXRbMTgwOjI1MF0sbG9uWzE5MjoyNzJd [following]
--2023-03-03 14:45:36-- https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fgoldsmr4.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNC5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9vcGVuZGFwL01FUlJBMl9ESVVSTkFML00yVFVOWEFERy41LjEyLjQvMjAwNi9NRVJSQTJfMzAwLnRhdmdVXzJkX2FkZ19OeC4yMDA2MDIubmM0Lm5jND9CQ0RQMDAxWzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLEJDRFAwMDJbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sQkNXVDAwMVswOjIzXVsxODA6MjUwXVsxOTI6MjcyXSxCQ1dUMDAyWzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLERVRFAwMDFbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sRFVEUDAwMlswOjIzXVsxODA6MjUwXVsxOTI6MjcyXSxEVURQMDAzWzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLERVRFAwMDRbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sRFVEUDAwNVswOjIzXVsxODA6MjUwXVsxOTI6MjcyXSxEVVdUMDAxWzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLERVV1QwMDJbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sRFVXVDAwM1swOjIzXVsxODA6MjUwXVsxOTI6MjcyXSxEVVdUMDA0WzA6MjNdWzE4MDoyNTBdWzE5MjoyNzJdLERVV1QwMDVbMDoyM11bMTgwOjI1MF1bMTkyOjI3Ml0sdGltZSxsYXRbMTgwOjI1MF0sbG9uWzE5MjoyNzJd
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
Unable to establish SSL connection.
I can download the individual files manually but it takes quite a long time due to the number of files. Any help would be greatly appreciated!

by **GES DISC - mgreene** » Mon Mar 06, 2023 12:54 pm America/New_York

Check if you can download the file https://goldsmr4.gesdisc.eosdis.nasa.gov/data/MERRA2_MONTHLY/M2TMNXRAD.5.12.4/1980/MERRA2_100.tavgM_2d_rad_Nx.198001.nc4 from the Web browser on your Windows desktop, if you do, then verify that you enter the correct password in wget prompt.
If the above steps check out then most likely the wget is outdated, check its version and see if you can install wget 1.21.

Data access instruction can be found here, https://disc.gsfc.nasa.gov/data-access

michelleescobarcarias01 · by **michelleescobarcarias01** » Mon Mar 06, 2023 9:19 pm America/New_York

I am having a similar issue but in my case, I first run into the following error message:
wget: unrecognised option `--ask-password'

The command I'm using is:
wget --load-cookies C:\Users\mesc0003\MERRA\.urs_cookies --save-cookies C:\Users\mesc0003\MERRA\.urs_cookies --auth-no-challenge=on --keep-session-cookies --user=michelleescobarcarias01 --ask-password --content-disposition -i \\ad.monash.edu\home\User042\mesc0003\Documents\NASA\nasa_humidity.txt

Any help would be highly appreciated

michelleescobarcarias01 · by **michelleescobarcarias01** » Mon Mar 06, 2023 11:07 pm America/New_York

So, I managed to solve the issue by updating my wget version but now, when I run the code only 2 documents are downloaded: the readme.pdf and the first link (day) of data. Then it stops as if it is done. Any clues?

by **GES DISC - mgreene** » Tue Mar 07, 2023 4:35 pm America/New_York

Re: wget Unable to establish SSL connection for MERRA2 data

by michelleescobarcarias01 » Mon Mar 06, 2023 11:07 pm America/New_York
So, I managed to solve the issue by updating my wget version but now, when I run the code only 2 documents are downloaded: the readme.pdf and the first link (day) of data. Then it stops as if it is done. Any clues?
Top

There was a server upgrade that may have cause this to happen. Please try again and let us know the status. thanks

michelleescobarcarias01 · by **michelleescobarcarias01** » Tue Mar 07, 2023 6:01 pm America/New_York

Hi,

I tried again several times but the same keeps happening.
Any recommendations?

Thanks!

Michelle

by **GES DISC - mgreene** » Thu Mar 09, 2023 10:12 pm America/New_York

Could you please provide us with the wget version you use now. We would also like for you to try the following
command, thanks:

wget --load-cookies C:\.urs_cookies --save-cookies C:\.urs_cookies
--auth-no-chal
lenge=on --keep-session-cookies --user=michelleescobarcarias01 --ask-password
--content-disposition
"https://goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2_DIURNAL/M2TUNXADG.5.12.4/2006/MERRA2_300.tavgU_2d_adg_Nx.200602.nc4.nc4?BCDP001[0:23][180:250][192:272],BCDP002[0:23][180:250][192:272],BCWT001[0:23][180:250][192:272],BCWT002[0:23][180:250][192:272],DUDP001[0:23][180:250][192:272],DUDP002[0:23][180:250][192:272],DUDP003[0:23][180:250][192:272],DUDP004[0:23][180:250][192:272],DUDP005[0:23][180:250][192:272],DUWT001[0:23][180:250][192:272],DUWT002[0:23][180:250][192:272],DUWT003[0:23][180:250][192:272],DUWT004[0:23][180:250][192:272],DUWT005[0:23][180:250][192:272],time,lat[180:250],lon[192:272]"

If the one-line command work, then put this command line into a loop to loop
through a time range using a programming language such as cshell.

by **GES DISC - mgreene** » Tue Mar 28, 2023 9:10 am America/New_York

Following up to see if the information provided met your needs.

cvwilson · by **cvwilson** » Mon Nov 06, 2023 11:48 am America/New_York

Hi, I am having the same issue but when I tried updating my version of wget to 1.21.4 this did not solve the issue. I'm able to download the link check you posted above in the browser using my login/password, but when I go to the command line I get the "Unable to establish SSL connection" error. Do you have more guidance?

by **GES DISC - mgreene** » Mon Nov 13, 2023 4:51 pm America/New_York

please check your wget version, 'wget -V'. It needs to be 1.20 or higher. If wget is compiled with older SSL package you might need to reinstall it.
First, try to download a single file first from your Web browser to verify the username/password and check if the .netrc file contains correct credentials:
https://goldsmr4.gesdisc.eosdis.nasa.gov/data/MERRA2/M2T1NXOCN.5.12.4/2019/01/MERRA2_400.tavg1_2d_ocn_Nx.20190102.nc4

Then, run wget for this file:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies https://goldsmr4.gesdisc.eosdis.nasa.gov/data/MERRA2/M2T1NXOCN.5.12.4/2019/01/MERRA2_400.tavg1_2d_ocn_Nx.20190102.nc4

Then, check your wget version, 'wget -V'. It needs to be 1.20 or higher. If wget is compiled with older SSL package you might need to reinstall it.

When you run wget to get subsets you will need to add --content-disposition option.