Earthdata Forum

Hi,I tried:wget --user=user --ask-password https://oceandata.sci.gsfc.nasa.gov (/cgi/getfile/A2019094102000.L2_LAC_OC.nc) but I get this error:--2020-01-15 15:17:13--  https://oceandata.sci.gsfc.nasa.gov (/cgi/getfile/A2019094102000.L2_LAC_OC.nc) Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0128::84Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.HTTP request sent, awaiting response... 302 FoundLocation: /ob/getfile/A2019094102000.L2_LAC_OC.nc [following]--2020-01-15 15:18:03--  https://oceandata.sci.gsfc.nasa.gov (/ob/getfile/A2019094102000.L2_LAC_OC.nc) Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.HTTP request sent, awaiting response... 403 Forbidden2020-01-15 15:18:03 ERROR 403: Forbidden.Did I do something wrong?Do you confirm that I can use wget --password to download in batch? (I didn't find this in https://oceancolor.gsfc.nasa.gov/data/download_methods/)Also, in some of your examples I see "echo" instead of "cgi" in the adress, is it normal?Thanks,Julien

I am having the same problem.  I added the --user and --password parameters to my wget command two days ago.  All was working fine until this morning. Now I am getting "ERROR 403 Forbidden" for all of my subscription downloads (L2 VIIRS, L2 MODIS & L3 MODIS).  Please advise.

Mark

I'm guessing this is unintended effect of the changeover to requiring username/password and OBPG already knows about it - but just in case they don't...

Since some time around 8 am EST we have been getting "403 Forbidden" in response to every request using the getfile CGI (https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/)

This is true both with and without known valid EarthData credentials being passed to wget using --user and --password

Oddly the file search cgi (https://oceandata.sci.gsfc.nasa.gov/api/file_search) seems to be working with and without the username/password.

If I can be of any help in testing/debugging changes please let me know.

-Andrew L.

Hi,

Currently, you need to use a curl command and a .netrc file to store your credentials. First go to "Options for passing credentials" under the cURL heading on the Data Download Methods page. Then use a command like:

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2019094102000.L2_LAC_OC.nc

to download your data. It should work.

We are investigating the cause and solution to the wget issues. Stay tuned for that outcome.

Similar issues here with downloads using curl. These worked yesterday (with UN/PW), but not today:

>curl -u UN:PW --retry 5 --retry-delay 2 --max-time 0 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2002335084000.L0_LAC.bz2

[no stdout, no file]

Side note: the call as above (using ~/.netrc and ~/.urs_cookies) DOES work...

Connection issues, however, seem to be affecting ancillary downloads:

> modis_atteph.py -s 2020014194500 -m aqua --ancdb=./anc.db --verbose

Searching database: ./anc.db
()
Input file: None
Sensor    : aqua
Start time: 2020014194500
End time  : None
()
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014/PM1EPHND_NRT.A2020014.1950.061
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014/PM1EPHND_NRT.A2020014.1945.061
  Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014/PM1EPHND_NRT.A2020014.1940.061
Downloading 'PM1ATTNR.P2020014.1800.003' to /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection error, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 503 - Service Temporarily Unavailable
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1ATTNR.P2020014.1800.003
***
*** Also check to make sure you have write permissions under the directory:
*** /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014
()

> getanc.py -s 2001168185000 --ancdb=./anc.db --verbose

Searching database: ./anc.db
()
Input file: None
Sensor    : None
Start time: 2001168185000
End time  : None
()
Downloading 'N200116900_MET_NCEPR2_6h.hdf' to /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2001/169
Connection interrupted, retrying up to 5 more time(s)
Connection error, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection error, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 503 - Service Temporarily Unavailable
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/N200116900_MET_NCEPR2_6h.hdf
***
*** Also check to make sure you have write permissions under the directory:
*** /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2001/169
()

In order to simplify things to try and diagnose, I tried a simple wget for the level 3 chlor product:

Command:
wget -t 3 --user=oceani --password='password' https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc

Result:
--2020-01-15 09:32:15--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc
Resolving oceandata.sci.gsfc.nasa.gov... xx.xxx.xx.xx
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc [following]
--2020-01-15 09:32:15--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&required_scope=study_area,country [following]
--2020-01-15 09:32:16--  https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: "A2020013.L3m_DAY_CHL_chlor_a_4km.nc"

    [ <=>                                                         ] 12,736      --.-K/s   in 0s     

    2020-01-15 09:32:16 (42.1 MB/s) -A2020013.L3m_DAY_CHL_chlor_a_4km.nc" saved [12736]

A listing of the file reveals that a file did get transferred. but not the NetCDF file reuqested:

ls -l A*
    -rw-rw-r--. 1 mark mark 12736 Jan 15 09:32 A2020013.L3m_DAY_CHL_chlor_a_4km.nc

...only 12,736 bytes.

I tried swapping the "cgi" out with "echo" in the URL as is shown in the "Download Methods" instructions and got teh same result.  I also put quotes around the URL with the same results. 

Any guidance/suggestions will be most appreciated.

Thank you,

Mark

I confirm that the curl+cookies method seems work.
But we use Python requests API to download the products, do you know how to setup cookies with these API?
Thanks

Tried curl + cookies and was able to retrieve a file, but it was only 665 bytes.  Standing by for word on a wget fix.

Thanks,

Mark

Mark,

Try adding "--auth-no-challenge=on" to your wget call, e.g.:

wget -t 3 --user=oceani --password='password' --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020013.L3m_DAY_CHL_chlor_a_4km.nc

Sean

That worked albeit it took some time and returned errors in the process. 

Command:
wget -t 3 --user=oceani --password='password' --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc 

Result:
--2020-01-15 11:07:05--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc
Resolving oceandata.sci.gsfc.nasa.gov... xx.xxx.xx.xx
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc [following]
--2020-01-15 11:07:06--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-01-15 11:07:06--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=dce33d17b8ed051e6d25336ea62c0e11e616f62dd6bde4e9f7fe2997fca018fc [following]
--2020-01-15 11:07:07--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=dce33d17b8ed051e6d25336ea62c0e11e616f62dd6bde4e9f7fe2997fca018fc
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc [following]
--2020-01-15 11:07:07--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020014.L3m_DAY_CHL_chlor_a_4km.nc
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8095505 (7.7M) [application/octet-stream]
Saving to: "A2020014.L3m_DAY_CHL_chlor_a_4km.nc"

100%[=====================================================>] 8,095,505   5.35M/s   in 1.4s   

2020-01-15 11:07:10 (5.35 MB/s) - "A2020014.L3m_DAY_CHL_chlor_a_4km.nc" saved [8095505/8095505]

I'll test this on my other subscription products and report back. 

Thanks,

Mark