Can't download GPM IMERG files in Windows 11 using wget

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
defendingfaithx
Posts: 4
Joined: Tue Feb 14, 2023 8:16 am America/New_York
Answers: 0

Can't download GPM IMERG files in Windows 11 using wget

by defendingfaithx » Tue Feb 14, 2023 8:53 am America/New_York

I'm getting a 'Username/Password Authentication Failed' and 'Unknown authentication scheme' error when trying to download GPM IMERG files through the OPeNDAP subsetting service using wget in Windows 11.

I've followed the 'wget for Windows' section in https://disc.gsfc.nasa.gov/data-access#windows_wget step-by-step and yet I continue to receive this error.

To start, launching cmd and making a cookies file results in a weird error. Following the steps in the above link, I use this code:
C:\Users\username\Desktop\folder>NUL > .urs_cookies

which gives me this error message:
'NUL' is not recognized as an internal or external command,
operable program or batch file.

However, the .urs_cookies file is still created and stored in the desktop folder. In fact, running the dir command in cmd (as per the instructions in this troubleshooting guide: https://daac.gsfc.nasa.gov/information/howto?title=How%20To%20resolve%20data%20download%20problems) for this folder's directory shows me that the file does exist (as a 0 byte URS_COOKIES file) and is stored in this folder.

Since the .urs_cookies file exists in the folder where I want to download the GPM IMERG files, I run the following code to download the GPM IMERG files through a .txt file with the file links (url.txt):
wget --load-cookies C:\Users\username\Desktop\folder\.urs_cookies --save-cookies C:\Users\username\Desktop\folder\.urs_cookies --auth-no-challenge=on --keep-session-cookies --user=<username> --ask-password --content-disposition -i url.txt

I'm able to download IMERG_ATBD_V06.pdf and README.GPM.pdf with no problems at all. After that however, it gives me the following error:
--2023-02-14 21:23:40-- https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHH.06/2020/299/3B-HHR.MS.MRG.3IMERG.20201025-S000000-E002959.0000.V06B.HDF5.nc4?precipitationCal[0:0][3023:3045][1017:1043],lat_bnds[1017:1043][0:1],lon_bnds[3023:3045][0:1],time,lon[3023:3045],lat[1017:1043],latv,lonv
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Unknown authentication scheme.

Username/Password Authentication Failed.
--2023-02-14 21:23:41-- https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHH.06/2020/299/3B-HHR.MS.MRG.3IMERG.20201025-S003000-E005959.0030.V06B.HDF5.nc4?precipitationCal[0:0][3023:3045][1017:1043],lat_bnds[1017:1043][0:1],lon_bnds[3023:3045][0:1],time,lon[3023:3045],lat[1017:1043],latv,lonv
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Unknown authentication scheme.

Username/Password Authentication Failed.
--2023-02-14 21:23:42-- https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHH.06/2020/299/3B-HHR.MS.MRG.3IMERG.20201025-S010000-E012959.0060.V06B.HDF5.nc4?precipitationCal[0:0][3023:3045][1017:1043],lat_bnds[1017:1043][0:1],lon_bnds[3023:3045][0:1],time,lon[3023:3045],lat[1017:1043],latv,lonv
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... connected.
Unable to establish SSL connection.
--2023-02-14 21:23:42-- https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHH.06/2020/299/3B-HHR.MS.MRG.3IMERG.20201025-S013000-E015959.0090.V06B.HDF5.nc4?precipitationCal[0:0][3023:3045][1017:1043],lat_bnds[1017:1043][0:1],lon_bnds[3023:3045][0:1],time,lon[3023:3045],lat[1017:1043],latv,lonv
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Unknown authentication scheme.

I think the root of the error could be in the creation of the .urs_cookies file but I am not so sure. I really have no idea how to fix this. Any help would be appreciated.

For reference, I'm using GNU wget 1.21.3 for Windows 11. A picture of the wget info, as well as the authentication error and the parameters of the GPM IMERG files I am trying to download are attached.
Attachments
wget info.png
wget info.png (115.3 KiB) Not viewed yet
authentication error.png
authentication error.png (157.75 KiB) Not viewed yet
imerg.png
imerg.png (41.78 KiB) Not viewed yet

Tags:

GES DISC - mgreene
User Services
User Services
Posts: 180
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Can't download GPM IMERG files in Windows 11 using wget

by GES DISC - mgreene » Tue Feb 14, 2023 10:10 am America/New_York

1. Could you please provide your Earthdata Login username, if not registered go to https://wiki.earthdata.nasa.gov/display/EL/How+To+Register+For+an+EarthData+Login+Profile
2. In your data download script replace --user=<username> with --user=YOUR_USERNAME in wget command,
3. Please run dir command in C:\Users\username\Desktop\folder> to see if .urs_cookies was created

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Can't download GPM IMERG files in Windows 11 using wget

by anthcook » Tue Feb 14, 2023 3:01 pm America/New_York

Hello,
I am a newbie user of wget with Windows, and after several trials and errors, this the the procedure that worked for me using wget on Windows.

1) Create a folder "zzzz", where "zzzz" is the folder name in a directory of your choice.
2) Copy cmd.exe, or some other equivalent of the a DOS console, into the "zzzz" folder.
3) Copy wget.exe into the "zzzz" folder.
4) Double-click cmd.exe to open the DOS console window.
5) Run the wget command at the DOS console:
wget -qO- --keep-session-cookies --save-cookies cookies.txt --post-data --user=YOURUSERNAME --ask-password https://urs.earthdata.nasa.gov/
, where YOURUSERNAME is your earthdata.nasa.gov username.
6) Enter your earthdata password when prompted.
7) This should have created a cookies.txt file. In my case, the cookies.txt file is place one directory above the "zzzz" folder.
8) My cookies file has contents that look something like this (if yours doesn't look like this, then perhaps something went wrong):

# HTTP Cookie File
# Generated by Wget on 2023-02-14 14:46:28.
# Edit at your own risk.

urs.earthdata.nasa.gov FALSE / FALSE 0 g6_afdg 1234
urs.earthdata.nasa.gov FALSE / FALSE 167980456 _urs-gui_session blah-blah-blah bunch of numbers and letters

9) Run the wget command:
wget --user=YOURUSERNAME --ask-password --auth-no-challenge=on URL_OF_DESIRED_DATA
, where YOURUSERNAME is your earthdata.nasa.gov username.
, where URL_OF_DESIRED_DATA = https://blahblahblah
In my case a typical URL would be something like https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc

Hope this helps,
Anthony

defendingfaithx
Posts: 4
Joined: Tue Feb 14, 2023 8:16 am America/New_York
Answers: 0

Re: Can't download GPM IMERG files in Windows 11 using wget

by defendingfaithx » Tue Feb 14, 2023 7:28 pm America/New_York

GES DISC - mgreene wrote:
> 1. Could you please provide your Earthdata Login username, if not registered go to
> https://wiki.earthdata.nasa.gov/display/EL/How+To+Register+For+an+EarthData+Login+Profile

My Earthdata Login name is defendingfaithx. I've also linked my Earthdata account to GES DISC a long time ago.

> 2. In your data download script replace --user=<username> with
> --user=YOUR_USERNAME in wget command,

I've been running the code with --user=defendingfaithx and the same error still occurs.

> 3. Please run dir command in C:\Users\username\Desktop\folder> to see if
> .urs_cookies was created

I run dir in C:\Users\username\Desktop\folder and it shows that the .urs_cookies file exists in the directory as a 0 byte URS_COOKIES file (see attached picture). However, the same error still occurs.
Attachments
dir.png
dir.png (53.5 KiB) Not viewed yet

defendingfaithx
Posts: 4
Joined: Tue Feb 14, 2023 8:16 am America/New_York
Answers: 0

Re: Can't download GPM IMERG files in Windows 11 using wget

by defendingfaithx » Tue Feb 14, 2023 8:42 pm America/New_York

anthcook wrote:
> Hello,
> I am a newbie user of wget with Windows, and after several trials and
> errors, this the the procedure that worked for me using wget on Windows.
>
> 1) Create a folder "zzzz", where "zzzz" is the folder
> name in a directory of your choice.
> 2) Copy cmd.exe, or some other equivalent of the a DOS console, into the
> "zzzz" folder.
> 3) Copy wget.exe into the "zzzz" folder.
> 4) Double-click cmd.exe to open the DOS console window.
> 5) Run the wget command at the DOS console:
> wget -qO- --keep-session-cookies --save-cookies cookies.txt --post-data
> --user=YOURUSERNAME --ask-password https://urs.earthdata.nasa.gov/
> , where YOURUSERNAME is your earthdata.nasa.gov username.
> 6) Enter your earthdata password when prompted.
> 7) This should have created a cookies.txt file. In my case, the
> cookies.txt file is place one directory above the "zzzz" folder.
> 8) My cookies file has contents that look something like this (if yours
> doesn't look like this, then perhaps something went wrong):
>
> # HTTP Cookie File
> # Generated by Wget on 2023-02-14 14:46:28.
> # Edit at your own risk.
>
> urs.earthdata.nasa.gov FALSE / FALSE 0 g6_afdg 1234
>
> urs.earthdata.nasa.gov FALSE / FALSE 167980456 _urs-gui_session blah-blah-blah
> bunch of numbers and letters
>
> 9) Run the wget command:
> wget --user=YOURUSERNAME --ask-password --auth-no-challenge=on
> URL_OF_DESIRED_DATA
> , where YOURUSERNAME is your earthdata.nasa.gov username.
> , where URL_OF_DESIRED_DATA = https://blahblahblah
> In my case a typical URL would be something like
> https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc
>
>
> Hope this helps,
> Anthony

Hi Anthony,

Thanks for the tips. I'm able to do them all successfully EXCEPT for step 9). I run the code exactly as you say:
wget --user=defendingfaithx --ask-password --auth-no-challenge=on https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHH.06/2020/299/3B-HHR.MS.MRG.3IMERG.20201025-S000000-E002959.0000.V06B.HDF5.nc4?precipitationCal[0:0][3023:3045][1017:1043],lat_bnds[1017:1043][0:1],lon_bnds[3023:3045][0:1],time,lon[3023:3045],lat[1017:1043],latv,lonv

and I get the same error:
Resolving gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)... 2001:4d0:241a:4041::50, 198.118.197.50
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|2001:4d0:241a:4041::50|:443... failed: Unknown error.
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Unknown authentication scheme.

Username/Password Authentication Failed.

I'm able to successfully create a cookies file with the contents you mentioned in step 8), though the error still persists.

Does this method work with getting file subsets? I did mention in my original post that I'm trying to get subsets from OPeNDAP.

Any other feedback would be appreciated.

anthcook
Posts: 20
Joined: Thu Feb 02, 2023 1:11 pm America/New_York
Answers: 0

Re: Can't download GPM IMERG files in Windows 11 using wget

by anthcook » Wed Feb 15, 2023 8:40 am America/New_York

Hello,
Here are a few troubleshooting tips to try:
1) See if you can download the file by pasting the long version (sub-setted) of the URL address below into a webpage browser:
https://gpm1.gesdisc.eosdis........latv,lonv

2) See if you can download the non-sub-setted version of file by pasting the short version of URL address into a webpage browser: https://gpm1.gesdisc.eosdis........V06B.HDF5.nc4

For me both (1) and (2) worked, and resulted in files of different sizes, as expected.

3) Try the wget download method for both the long (subsetted) and short(non-subsetted) filenames.

For me, both of these failed with a similar error as yours:

Resolving gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)... 198.118.197.50
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Unknown authentication scheme.
Username/Password Authentication Failed.

4) To check the integrity of your wget setup, try the wget download method for an earthdata file from a different instrument, such as:
wget --user=YOURUSERNAME --ask-password --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc

For me this worked.



The conclusion from the above steps, at least for me, and likely for you if you get the same results, is that:

a) The wget setup is working properly.

b) There are no issues with user authorization and downloading either subsetted or non-subsetted GPM_3IMERGHH files manually from the website browser.

c) ***There seems to be an authorization issue reaching the GPM_3IMERGHH files with the wget method. ***

d) Until (c) is resolved, no conclusion can be reached yet about the utility of downloading either subsetted or non-subsetted GPM_3IMERGHH files with wget.

If conclusion (c) is correct, then hopefully one of the earthdata staff members can shed more light on the situation.

Anthony

GES DISC - mgreene
User Services
User Services
Posts: 180
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Can't download GPM IMERG files in Windows 11 using wget

by GES DISC - mgreene » Wed Feb 15, 2023 8:46 am America/New_York

User, defendingfaithx
Please try the following

1. In your data download script replace --user=<username> with --user=YOUR_USERNAME in wget command,
2. Please run dir command in C:\Users\username\Desktop\folder> to see if .urs_cookies was created

defendingfaithx
Posts: 4
Joined: Tue Feb 14, 2023 8:16 am America/New_York
Answers: 0

Re: Can't download GPM IMERG files in Windows 11 using wget

by defendingfaithx » Wed Feb 15, 2023 8:55 pm America/New_York

anthcook wrote:
> Hello,
> Here are a few troubleshooting tips to try:
> 1) See if you can download the file by pasting the long version
> (sub-setted) of the URL address below into a webpage browser:
> https://gpm1.gesdisc.eosdis........latv,lonv
>
> 2) See if you can download the non-sub-setted version of file by pasting
> the short version of URL address into a webpage browser:
> https://gpm1.gesdisc.eosdis........V06B.HDF5.nc4
>
> For me both (1) and (2) worked, and resulted in files of different sizes,
> as expected.
>
> 3) Try the wget download method for both the long (subsetted) and
> short(non-subsetted) filenames.
>
> For me, both of these failed with a similar error as yours:
>
> Resolving gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)...
> 198.118.197.50
> Connecting to gpm1.gesdisc.eosdis.nasa.gov
> (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... connected.
> HTTP request sent, awaiting response... 401 Unauthorized
> Unknown authentication scheme.
> Username/Password Authentication Failed.
>
> 4) To check the integrity of your wget setup, try the wget download method
> for an earthdata file from a different instrument, such as:
> wget --user=YOURUSERNAME --ask-password --auth-no-challenge=on
> https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20220101.L3m.DAY.IOP.a_531.4km.nc
>
> For me this worked.
>
>
>
> The conclusion from the above steps, at least for me, and likely for you if
> you get the same results, is that:
>
> a) The wget setup is working properly.
>
> b) There are no issues with user authorization and downloading either
> subsetted or non-subsetted GPM_3IMERGHH files manually from the website
> browser.
>
> c) ***There seems to be an authorization issue reaching the GPM_3IMERGHH
> files with the wget method. ***
>
> d) Until (c) is resolved, no conclusion can be reached yet about the
> utility of downloading either subsetted or non-subsetted GPM_3IMERGHH files
> with wget.
>
> If conclusion (c) is correct, then hopefully one of the earthdata staff
> members can shed more light on the situation.
>
> Anthony

Hi Anthony,

Thanks for the really thorough tips, I appreciate it.

Step 1) and Step 2) both work for me. It's going to be a bit of a hassle to download hundreds of files this way, but it's better than nothing.

With that being said, Step 3) failed to download the subsetted files. I used the following code:
wget --load-cookies C:\.urs_cookies --save-cookies C:\.urs_cookies --auth-no-challenge=on --keep-session-cookies --user=defendingfaithx --ask-password --content-disposition -i filelist.txt

...which resulted in the following error:
Username/Password Authentication Failed.
The destination name is too long (248), reducing to 241
--2023-02-16 09:51:52-- https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHH.06/2020/299/3B-HHR.MS.MRG.3IMERG.20201025-S003000-E005959.0030.V06B.HDF5.nc4?precipitationCal[0:0][3023:3044][1017:1043],lat_bnds[1017:1043][0:1],time_bnds[0:0][0:1],lon_bnds[3023:3044][0:1],time,lon[3023:3044],lat[1017:1043],latv,nv,lonv
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... connected.
Unable to establish SSL connection.
The destination name is too long (248), reducing to 241
--2023-02-16 09:51:52-- https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGHH.06/2020/299/3B-HHR.MS.MRG.3IMERG.20201025-S010000-E012959.0060.V06B.HDF5.nc4?precipitationCal[0:0][3023:3044][1017:1043],lat_bnds[1017:1043][0:1],time_bnds[0:0][0:1],lon_bnds[3023:3044][0:1],time,lon[3023:3044],lat[1017:1043],latv,nv,lonv
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Unknown authentication scheme.

However, I was able to download non-subsetted files using the following code:
wget --load-cookies C:\.urs_cookies --save-cookies C:\.urs_cookies --auth-no-challenge=on --keep-session-cookies --user=<your username> --ask-password -i filelist.txt

Step 4) worked for me. I was able to download the file you linked with the code you sent.

With all this being said, perhaps there is an authorization issue reaching the GPM_3IMERGHH subsetted files using the wget method with Windows, as you suggested. I don't know if the issue is with GES DISC, OPeNDAP, or somewhere else. I wonder who from the staff can be approached for this issue.

Thanks very much Anthony. Feels good to see we're getting somewhere with this.

GES DISC - mgreene
User Services
User Services
Posts: 180
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Can't download GPM IMERG files in Windows 11 using wget

by GES DISC - mgreene » Thu Feb 16, 2023 10:05 am America/New_York

To User, defendingfaithx
Please try the following,

1. In your data download script replace --user=<username> with --user=YOUR_USERNAME in wget command,
2. Please run dir command in C:\Users\username\Desktop\folder> to see if .urs_cookies was created

johanbv
Posts: 1
Joined: Sat Mar 11, 2023 1:58 am America/New_York
Answers: 0

Re: Can't download GPM IMERG files in Windows 11 using wget

by johanbv » Sat Mar 11, 2023 2:02 am America/New_York

I have the exact same issue as described here, only for the GPM_3IMERGDF dataset compared to the GPM_3IMERGHH mentioned in this post. However, GPM_3IMERGDF is based off GPM_3IMERGHH data. I've been through all the troubleshooting steps as suggested by Anthony and I finding the same results as user defendingfaithx. It seems like there is an issue with authentication and these datasets when using wget, and it's not just the users entering their passwords incorrectly in the wget command.

Post Reply