wget EarthData Login does not work

Please enter here to ask a question about any NASA Science related topics!
dem1
Posts: 40
Joined: Mon Nov 28, 2005 4:49 am America/New_York

wget EarthData Login does not work

by dem1 » Wed Jan 15, 2020 9:21 am America/New_York

Hi,

I tried:
wget --user=user --ask-password https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019094102000.L2_LAC_OC.nc
but I get this error:
--2020-01-15 15:17:13--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019094102000.L2_LAC_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2019094102000.L2_LAC_OC.nc [following]
--2020-01-15 15:18:03--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2019094102000.L2_LAC_OC.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 403 Forbidden
2020-01-15 15:18:03 ERROR 403: Forbidden.

Did I do something wrong?
Do you confirm that I can use wget --password to download in batch? (I didn't find this in https://oceancolor.gsfc.nasa.gov/data/download_methods/)
Also, in some of your examples I see "echo" instead of "cgi" in the adress, is it normal?

Thanks,
Julien

Tags:

oceani
Posts: 33
Joined: Wed Aug 23, 2006 12:32 pm America/New_York

wget EarthData Login does not work

by oceani » Wed Jan 15, 2020 9:35 am America/New_York

I am having the same problem.  I added the --user and --password parameters to my wget command two days ago.  All was working fine until this morning. Now I am getting "ERROR 403 Forbidden" for all of my subscription downloads (L2 VIIRS, L2 MODIS & L3 MODIS).  Please advise.

Mark

alaroy
Posts: 57
Joined: Tue Nov 21, 2006 3:33 pm America/New_York

wget EarthData Login does not work

by alaroy » Wed Jan 15, 2020 10:32 am America/New_York

I'm guessing this is unintended effect of the changeover to requiring username/password and OBPG already knows about it - but just in case they don't...

Since some time around 8 am EST we have been getting "403 Forbidden" in response to every request using the getfile CGI (https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/)

This is true both with and without known valid EarthData credentials being passed to wget using --user and --password

Oddly the file search cgi (https://oceandata.sci.gsfc.nasa.gov/api/file_search) seems to be working with and without the username/password.

If I can be of any help in testing/debugging changes please let me know.

-Andrew L.

OB.DAAC - amscott
User Services
User Services
Posts: 64
Joined: Mon Jun 22, 2020 5:24 pm America/New_York

wget EarthData Login does not work

by OB.DAAC - amscott » Wed Jan 15, 2020 11:36 am America/New_York

Hi,

Currently, you need to use a curl command and a .netrc file to store your credentials. First go to "Options for passing credentials" under the cURL heading on the Data Download Methods page. Then use a command like:

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2019094102000.L2_LAC_OC.nc

to download your data. It should work.

We are investigating the cause and solution to the wget issues. Stay tuned for that outcome.

bbbarnes
Posts: 23
Joined: Tue Feb 04, 2014 12:57 pm America/New_York

wget EarthData Login does not work

by bbbarnes » Wed Jan 15, 2020 11:45 am America/New_York

Similar issues here with downloads using curl. These worked yesterday (with UN/PW), but not today:

>curl -u UN:PW --retry 5 --retry-delay 2 --max-time 0 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2002335084000.L0_LAC.bz2


[no stdout, no file]

Side note: the call as above (using ~/.netrc and ~/.urs_cookies) DOES work...

Connection issues, however, seem to be affecting ancillary downloads:

> modis_atteph.py -s 2020014194500 -m aqua --ancdb=./anc.db --verbose


Searching database: ./anc.db
()
Input file: None
Sensor    : aqua
Start time: 2020014194500
End time  : None
()
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014/PM1EPHND_NRT.A2020014.1950.061
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014/PM1EPHND_NRT.A2020014.1945.061
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014/PM1EPHND_NRT.A2020014.1940.061
Downloading 'PM1ATTNR.P2020014.1800.003' to /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection error, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 503 - Service Temporarily Unavailable
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1ATTNR.P2020014.1800.003
***
*** Also check to make sure you have write permissions under the directory:
*** /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014
()

> getanc.py -s 2001168185000 --ancdb=./anc.db --verbose


Searching database: ./anc.db
()
Input file: None
Sensor    : None
Start time: 2001168185000
End time  : None
()
Downloading 'N200116900_MET_NCEPR2_6h.hdf' to /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2001/169
Connection interrupted, retrying up to 5 more time(s)
Connection error, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection error, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 503 - Service Temporarily Unavailable
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/N200116900_MET_NCEPR2_6h.hdf
***
*** Also check to make sure you have write permissions under the directory:
*** /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2001/169
()

oceani
Posts: 33
Joined: Wed Aug 23, 2006 12:32 pm America/New_York

wget EarthData Login does not work

by oceani » Wed Jan 15, 2020 11:48 am America/New_York

In order to simplify things to try and diagnose, I tried a simple wget for the level 3 chlor product:

Command:
wget -t 3 --user=oceani --password='password' https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc

Result:
--2020-01-15 09:32:15--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc
Resolving oceandata.sci.gsfc.nasa.gov... xx.xxx.xx.xx
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc [following]
--2020-01-15 09:32:15--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&required_scope=study_area,country [following]
--2020-01-15 09:32:16--  https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: "A2020013.L3m_DAY_CHL_chlor_a_4km.nc"

    [ <=>                                                         ] 12,736      --.-K/s   in 0s     

    2020-01-15 09:32:16 (42.1 MB/s) -A2020013.L3m_DAY_CHL_chlor_a_4km.nc" saved [12736]

A listing of the file reveals that a file did get transferred. but not the NetCDF file reuqested:

ls -l A*
    -rw-rw-r--. 1 mark mark 12736 Jan 15 09:32 A2020013.L3m_DAY_CHL_chlor_a_4km.nc

...only 12,736 bytes.

I tried swapping the "cgi" out with "echo" in the URL as is shown in the "Download Methods" instructions and got teh same result.  I also put quotes around the URL with the same results. 

Any guidance/suggestions will be most appreciated.

Thank you,

Mark

dem1
Posts: 40
Joined: Mon Nov 28, 2005 4:49 am America/New_York

wget EarthData Login does not work

by dem1 » Wed Jan 15, 2020 11:54 am America/New_York

I confirm that the curl+cookies method seems work.
But we use Python requests API to download the products, do you know how to setup cookies with these API?
Thanks

oceani
Posts: 33
Joined: Wed Aug 23, 2006 12:32 pm America/New_York

wget EarthData Login does not work

by oceani » Wed Jan 15, 2020 12:22 pm America/New_York

Tried curl + cookies and was able to retrieve a file, but it was only 665 bytes.  Standing by for word on a wget fix.

Thanks,

Mark

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1167
Joined: Wed Sep 18, 2019 6:15 pm America/New_York

wget EarthData Login does not work

by OB.DAAC - SeanBailey » Wed Jan 15, 2020 12:51 pm America/New_York

Mark,

Try adding "--auth-no-challenge=on" to your wget call, e.g.:

wget -t 3 --user=oceani --password='password' --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc

Sean

oceani
Posts: 33
Joined: Wed Aug 23, 2006 12:32 pm America/New_York

wget EarthData Login does not work

by oceani » Wed Jan 15, 2020 1:17 pm America/New_York

That worked albeit it took some time and returned errors in the process. 

Command:
wget -t 3 --user=oceani --password='password' --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc 

Result:
--2020-01-15 11:07:05--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc
Resolving oceandata.sci.gsfc.nasa.gov... xx.xxx.xx.xx
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc [following]
--2020-01-15 11:07:06--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-01-15 11:07:06--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=dce33d17b8ed051e6d25336ea62c0e11e616f62dd6bde4e9f7fe2997fca018fc [following]
--2020-01-15 11:07:07--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=dce33d17b8ed051e6d25336ea62c0e11e616f62dd6bde4e9f7fe2997fca018fc
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc [following]
--2020-01-15 11:07:07--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8095505 (7.7M) [application/octet-stream]
Saving to: "A2020014.L3m_DAY_CHL_chlor_a_4km.nc"

100%[=====================================================>] 8,095,505   5.35M/s   in 1.4s   

2020-01-15 11:07:10 (5.35 MB/s) - "A2020014.L3m_DAY_CHL_chlor_a_4km.nc" saved [8095505/8095505]

I'll test this on my other subscription products and report back. 

Thanks,

Mark

Post Reply