Authentication failure - using wget via HPC based Linux

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Authentication failure - using wget via HPC based Linux

by pmamegbor » Wed Apr 19, 2023 2:05 pm America/New_York

Hi

I am trying to download M2I3NVAER .nc data from GES DISC, using wget and following the instructions here.

Due to the size of the data over 1TB I am downloading the data to HPC platform. I have the correct login details pr credentials. However, I get a username/password failure with the following message

Reusing existing connection to urs.earthdata.nasa.gov:443.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.
--2023-04-19 13:54:46-- https://goldsmr5.gesdisc.eosdis.nasa.gov/daac-bin/OTF/HTTP_services.cgi?FILENAME=%2Fdata%2FMERRA2%2FM2I3NVAER.5.12.4%2F2021%2F12%2FMERRA2_400.inst3_3d_aer_Nv.20211230.nc4&SERVICE=L34RS_MERRA2&SHORTNAME=M2I3NVAER&VERSION=1.02&BBOX=-90%2C-180%2C90%2C180&FORMAT=bmM0Lw&DATASET_VERSION=5.12.4&LABEL=MERRA2_400.inst3_3d_aer_Nv.20211230.SUB.nc&VARIABLES=AIRDENS%2CBCPHILIC%2CBCPHOBIC%2CDU001%2CDU002%2CDU003%2CDU004%2CDU005%2COCPHILIC%2COCPHOBIC%2CSO4%2CSS001%2CSS002%2CSS003%2CSS004%2CSS005
Connecting to goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)|198.118.197.96|:443... connected.
HTTP request sent, awaiting response... 302 Found
The name is too long, 395 chars total.
Trying to shorten...

Any advice on how to resolve this issues would be greatly appreciated. Thank you

Tags:

GES DISC - mgreene
User Services
User Services
Posts: 191
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Wed Apr 19, 2023 3:57 pm America/New_York

This is the GES DISC User Service. First please follow the data access instruction in the link below,
https://disc.gsfc.nasa.gov/data-access Make sure you have a wget of 1.18 or greater.

pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Re: Authentication failure - using wget via HPC based Linux

by pmamegbor » Wed Apr 19, 2023 4:21 pm America/New_York

Hi,
I did follow the instruction on https://disc.gsfc.nasa.gov/data-access

I have used the instruction to access other datasets e.g. SO2, organic carbon, etc. This is the first time I am doing so via an HPC platform.

The wget information shows version 1.19 (GNU Wget 1.19.5 built on linux-gnu)

GES DISC - mgreene
User Services
User Services
Posts: 191
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Thu Apr 20, 2023 10:22 am America/New_York

Could you please provided the full wget command you are issuing.

pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Re: Authentication failure - using wget via HPC based Linux

by pmamegbor » Thu Apr 20, 2023 1:42 pm America/New_York

This is the command I used

cd $/scratch/xxxx/
touch .netrc
echo "machine urs.earthdata.nasa.gov login xxxx password xxxx" >> .netrc
chmod 0600 .netrc
touch .urs_cookies.

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition -i PM_Data_2020-21.txt (using a subset of the M2I3NVAER data, subsetted by the years in the file name and variables for computing PM10)

GES DISC - mgreene
User Services
User Services
Posts: 191
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Thu Apr 20, 2023 2:52 pm America/New_York

thank you

GES DISC - mgreene
User Services
User Services
Posts: 191
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Thu Apr 20, 2023 3:47 pm America/New_York

Please remove '--auth-no-challenge=on' option from wget

Also .netrc has to be in home directory -- looks like you created it in the directory
other than home

GES DISC - mgreene
User Services
User Services
Posts: 191
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Authentication failure - using wget via HPC based Linux

by GES DISC - mgreene » Tue Apr 25, 2023 10:05 am America/New_York

Following up to my last post to see if you found the information to be helpful. thanks

pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Re: Authentication failure - using wget via HPC based Linux

by pmamegbor » Tue May 02, 2023 1:20 pm America/New_York

Sorry, for the late reply. I got caught up in other duties but unfortunately, it did not work. Also, I don't think the problem is not using the home directory. I have accessed and downloaded other GES DISC data e.g. NO2 on folders other than the home directory. I downloaded the other data to my external hard drive following the same instructions here: https://disc.gsfc.nasa.gov/data-access. I am using the HPC platform due to the size of the M2I3NVAER .nc data

I tried downloading just a day's data to my external hard drive. It seems to be working but I get

Connecting to goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)|95.100.168.66|:443... failed: Connection timed out.

pmamegbor
Posts: 11
Joined: Wed Apr 19, 2023 1:55 pm America/New_York
Answers: 0

Re: Authentication failure - using wget via HPC based Linux

by pmamegbor » Tue May 02, 2023 1:26 pm America/New_York

GES DISC - mgreene wrote:
> Following up to my last post to see if you found the information to be
> helpful. thanks
It failed with the following output


--2023-05-02 13:24:26-- https://goldsmr5.gesdisc.eosdis.nasa.gov/daac-bin/OTF/HTTP_services.cgi?FILENAME=%2Fdata%2FMERRA2%2FM2I3NVAER.5.12.4%2F2023%2F03%2FMERRA2_400.inst3_3d_aer_Nv.20230331.nc4&VARIABLES=AIRDENS%2CBCPHILIC%2CBCPHOBIC%2CDU001%2CDU002%2CDU003%2CDU004%2CDU005%2COCPHILIC%2COCPHOBIC%2CSO2%2CSO4%2CSS001%2CSS002%2CSS003%2CSS004%2CSS005&VERSION=1.02&LABEL=MERRA2_400.inst3_3d_aer_Nv.20230331.SUB.nc&DATASET_VERSION=5.12.4&SHORTNAME=M2I3NVAER&BBOX=-90%2C-180%2C90%2C180&FORMAT=bmM0Lw&SERVICE=L34RS_MERRA2
Connecting to goldsmr5.gesdisc.eosdis.nasa.gov (goldsmr5.gesdisc.eosdis.nasa.gov)|198.118.197.96|:443... connected.
HTTP request sent, awaiting response... 302 Found
The destination name is too long (399), reducing to 236
Location: https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=https%3A%2F%2Fgoldsmr5.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYWFjLWJpbi9PVEYvSFRUUF9zZXJ2aWNlcy5jZ2k%2FRklMRU5BTUU9JTJGZGF0YSUyRk1FUlJBMiUyRk0ySTNOVkFFUi41LjEyLjQlMkYyMDIzJTJGMDMlMkZNRVJSQTJfNDAwLmluc3QzXzNkX2Flcl9Odi4yMDIzMDMzMS5uYzQmVkFSSUFCTEVTPUFJUkRFTlMlMkNCQ1BISUxJQyUyQ0JDUEhPQklDJTJDRFUwMDElMkNEVTAwMiUyQ0RVMDAzJTJDRFUwMDQlMkNEVTAwNSUyQ09DUEhJTElDJTJDT0NQSE9CSUMlMkNTTzIlMkNTTzQlMkNTUzAwMSUyQ1NTMDAyJTJDU1MwMDMlMkNTUzAwNCUyQ1NTMDA1JlZFUlNJT049MS4wMiZMQUJFTD1NRVJSQTJfNDAwLmluc3QzXzNkX2Flcl9Odi4yMDIzMDMzMS5TVUIubmMmREFUQVNFVF9WRVJTSU9OPTUuMTIuNCZTSE9SVE5BTUU9TTJJM05WQUVSJkJCT1g9LTkwJTJDLTE4MCUyQzkwJTJDMTgwJkZPUk1BVD1ibU0wTHcmU0VSVklDRT1MMzRSU19NRVJSQTI [following]
--2023-05-02 13:24:26-- https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=https%3A%2F%2Fgoldsmr5.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9kYWFjLWJpbi9PVEYvSFRUUF9zZXJ2aWNlcy5jZ2k%2FRklMRU5BTUU9JTJGZGF0YSUyRk1FUlJBMiUyRk0ySTNOVkFFUi41LjEyLjQlMkYyMDIzJTJGMDMlMkZNRVJSQTJfNDAwLmluc3QzXzNkX2Flcl9Odi4yMDIzMDMzMS5uYzQmVkFSSUFCTEVTPUFJUkRFTlMlMkNCQ1BISUxJQyUyQ0JDUEhPQklDJTJDRFUwMDElMkNEVTAwMiUyQ0RVMDAzJTJDRFUwMDQlMkNEVTAwNSUyQ09DUEhJTElDJTJDT0NQSE9CSUMlMkNTTzIlMkNTTzQlMkNTUzAwMSUyQ1NTMDAyJTJDU1MwMDMlMkNTUzAwNCUyQ1NTMDA1JlZFUlNJT049MS4wMiZMQUJFTD1NRVJSQTJfNDAwLmluc3QzXzNkX2Flcl9Odi4yMDIzMDMzMS5TVUIubmMmREFUQVNFVF9WRVJTSU9OPTUuMTIuNCZTSE9SVE5BTUU9TTJJM05WQUVSJkJCT1g9LTkwJTJDLTE4MCUyQzkwJTJDMTgwJkZPUk1BVD1ibU0wTHcmU0VSVklDRT1MMzRSU19NRVJSQTI
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 95.100.168.66, 2.16.40.66, ...
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.
FINISHED --2023-05-02 13:24:26--
Total wall clock time: 13m 22s
Downloaded: 1 files, 1.0M in 0.07s (13.9 MB/s)

Post Reply