Non-interactive downlaods through Earthdata login

Please enter here to ask a question about any NASA Science related topics!
Post Reply
b.mccabe
Posts: 20
Joined: Mon Oct 13, 2014 1:18 pm America/New_York

Non-interactive downlaods through Earthdata login

by b.mccabe » Wed Nov 20, 2019 4:36 pm America/New_York

With regards to the recently announced move the requiring EarthData login for data downloads, we have been unable to make non-interactive downloads work using the guidance at https://oceancolor.gsfc.nasa.gov/data/download_methods/.  Using a HICO file as a test, as it is already requires EarthData login, attempting to download the file results in the source code for the login page being retrieved instead:

[bmccabe1@sora ~]$ wget --user=b.mccabe --password='mypassword' https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
--2019-11-20 21:17:41--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Nwx7SywSELFPjkOk6Ifsjw&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict [following]
--2019-11-20 21:17:42--  https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Nwx7SywSELFPjkOk6Ifsjw&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘H2014255140230.L2_ISS_OC.nc’

    [ <=>                                                                                                                                      ] 12,663      --.-K/s   in 0.001s 

2019-11-20 21:17:42 (14.9 MB/s) - ‘H2014255140230.L2_ISS_OC.nc’ saved [12663]

[bmccabe1@sora ~]$ head -n 10 H2014255140230.L2_ISS_OC.nc
<!DOCTYPE html>
<!--[if lt IE 7]><html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]><html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]><html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--><html lang="en" class="no-js"><!--<![endif]-->
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
    <title>Earthdata Login</title>
    <meta name="description" content="Earthdata Login">


Attempting to store my credentials via the ~/.netrc file and calling wget with a cookie fares no better:

[bmccabe1@sora ~]$ rm H2014255140230.L2_ISS_OC.nc
[bmccabe1@sora ~]$ echo 'machine urs.earthdata.nasa.gov login b.mccabe password mypassword' > ~/.netrc
[bmccabe1@sora ~]$ chmod 0600 ~/.netrc
[bmccabe1@sora ~]$ touch ~/.urs_cookies
[bmccabe1@sora ~]$ wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
--2019-11-20 21:23:32--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw [following]
--2019-11-20 21:23:32--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘H2014255140230.L2_ISS_OC.nc’

    [ <=>                                                                                                                                      ] 12,663      --.-K/s   in 0.001s 

2019-11-20 21:23:32 (14.6 MB/s) - ‘H2014255140230.L2_ISS_OC.nc’ saved [12663]

[bmccabe1@sora ~]$ head -n 10 H2014255140230.L2_ISS_OC.nc
<!DOCTYPE html>
<!--[if lt IE 7]><html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]><html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]><html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--><html lang="en" class="no-js"><!--<![endif]-->
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
    <title>Earthdata Login</title>
    <meta name="description" content="Earthdata Login">


Using curl yields similar results.  I am able to successfully download HICO files using the web browser.  Any ideas why I am unable to successfully download a file with Earthdata login non-interactively?

Thanks,
Benjamin McCabe/Computer Scientist, Direct Readout Laboratory
NASA/Goddard Space Flight Center
http://directreadout.sci.gsfc.nasa.gov/
https://www.facebook.com/directreadout/
https://twitter.com/NASA_DRL

Tags:

b.mccabe
Posts: 20
Joined: Mon Oct 13, 2014 1:18 pm America/New_York

Non-interactive downlaods through Earthdata login

by b.mccabe » Tue Dec 03, 2019 3:45 pm America/New_York

Any ideas why our download attempts are resulting in us downloading the login page rather than the target file?  We very much would like to have our applications updated to be able to continue retrieving the ancillary files we need before this change goes into effect.

Thanks,
Benjamin McCabe/Computer Scientist, Direct Readout Laboratory
NASA/Goddard Space Flight Center
http://directreadout.sci.gsfc.nasa.gov/
https://www.facebook.com/directreadout/
https://twitter.com/NASA_DRL

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1195
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

Non-interactive downlaods through Earthdata login

by OB.DAAC - SeanBailey » Tue Dec 03, 2019 6:03 pm America/New_York

Hmmm...wget gives me the same problem, but curl works:

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc

We'll dig into why the wget behavior changed (it worked for me when we wrote the original tutorial several year ago).  In the meantime, give curl a try and see if that will work for you.

Sean

b.mccabe
Posts: 20
Joined: Mon Oct 13, 2014 1:18 pm America/New_York

Non-interactive downlaods through Earthdata login

by b.mccabe » Wed Dec 04, 2019 2:22 pm America/New_York

Thanks.  I was able to perform a successful download using curl, and look forward to hearing your findings regarding wget.

Thanks,
Benjamin McCabe/Computer Scientist, Direct Readout Laboratory
NASA/Goddard Space Flight Center
http://directreadout.sci.gsfc.nasa.gov/
https://www.facebook.com/directreadout/
https://twitter.com/NASA_DRL

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1195
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

Non-interactive downlaods through Earthdata login

by OB.DAAC - SeanBailey » Fri Dec 06, 2019 2:36 pm America/New_York

Benjamin,
Might have been a wget version problem - seems there was an issue with .netrc implementation pre 1.19.1

Using wget versions 1.19.4 (Ubuntu 18.04 Linux) and 1.20.3 (Mac), the following worked for me:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --no-check-certificate https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc

...yes pretty much the previous example.

Sean

b.mccabe
Posts: 20
Joined: Mon Oct 13, 2014 1:18 pm America/New_York

Non-interactive downlaods through Earthdata login

by b.mccabe » Tue Dec 10, 2019 2:52 pm America/New_York

Perhaps, but unfortunately wget 1.14 is the latest available through the CentOS 7 package repository. Also, I note that my first command:
wget --user=b.mccabe --password='mypassword' https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
Does not work despite not using .netrc.

Thanks,
Ben

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 160
Joined: Fri Feb 05, 2021 9:17 am America/New_York

Non-interactive downlaods through Earthdata login

by OB ODPS - towens » Tue Dec 10, 2019 3:46 pm America/New_York

Can you build wget-1.20 from source?  https://ftp.gnu.org/gnu/wget/

Tommy

b.mccabe
Posts: 20
Joined: Mon Oct 13, 2014 1:18 pm America/New_York

Non-interactive downlaods through Earthdata login

by b.mccabe » Mon Dec 16, 2019 3:39 pm America/New_York

We would rather not move away from the package manager managed version of the software on this system if at all possible, especially since we have other applications which are successfully using the existing wget version which could potentially be impacted by an upgrade.

Thanks,
Ben

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1195
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

Non-interactive downlaods through Earthdata login

by OB.DAAC - SeanBailey » Mon Dec 16, 2019 7:01 pm America/New_York

Ben,

Understandable. 

Try adding --auth-no-challenge to the command line options for wget.  I found a machine with 1.14 on it and that seemed to do the trick, e.g.:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc

...or use cURL...that also worked for me on that Centos7 machine :grin:

Sean

Post Reply