wget EarthData Login does not work

oceani · by **oceani** » Wed Jan 15, 2020 1:41 pm America/New_York

Sean,

The --auth-no-challenge=on addition seems to be working for our other subscription file requests/downloads. The server does, however, return a series of "HTTP request sent, awaiting response... 302 Found" messages for every file requested for download. Not necessarily a serious problem, but does increase the file retrieval time significantly as there is a 1-3 minute delay for each file during this process. Thanks for the help. If you have any additional suggestions to get rid of the delay, let me know.

Thanks again,

Mark

by **OB.DAAC - SeanBailey** » Wed Jan 15, 2020 1:42 pm America/New_York

Mark,

I don't see any errors in the output...Sure it's blabbing on about redirection, but those are indeed the magic that makes the authentication work, and so are expected. You can tell wget to be quiet with the "-q" option.

You should also try using the .netrc approach (see https://oceancolor.gsfc.nasa.gov/data/download_methods/), it's a wee bit more secure as you don't need to send the credentials in plain text.

Sean

oceani · by **oceani** » Wed Jan 15, 2020 1:57 pm America/New_York

Good enough. I guess the extra delay in file retrieval is just now part of the new authentication protocol. We can live with it. Will certainly set up the .netrc method.... in time when we can get to it.

Thanks,

Mark

oo_processing · by **oo_processing** » Wed Jan 15, 2020 3:23 pm America/New_York

While it appears that the .netrc method is working for specific files, I am still having issues with OCSSW scripts like this (nothing else is running):

$modis_atteph_command:/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/scripts/modis_atteph.py -m aqua -s 2020014194502 -e 2020014194502 --verbose --ancdb=./ancillary_data.db
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/scripts/modis_atteph.py -m aqua -s 2020014194502 -e 2020014194502 --verbose --ancdb=./ancillary_data.db returns: Searching database: ./ancillary_data.db
()
Input file: None
Sensor : aqua
Start time: 2020014194502
End time : 2020014194502
()
Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014/PM1EPHND_NRT.A2020014.1945.061
Found: /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014/PM1EPHND_NRT.A2020014.1940.061
Downloading 'PM1ATTNR.P2020014.1800.003' to /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
URL attempted: oceandata.sci.gsfc.nasa.gov
Well, this is embarrassing...an error occurred that we just cannot get past...
Here is what we know: ('The read operation timed out',)
Please retry this request at a later time.
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1ATTNR.P2020014.1800.003
***
*** Also check to make sure you have write permissions under the directory:
*** /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/014
()

ionutserban · by **ionutserban** » Wed Jan 15, 2020 3:49 pm America/New_York

Hi!

I had the same issue today, a few days ago the same command worked perfectly.

I just want to add one thing. If I try "wget oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1ATTNR.P2020014.1800.003", the file can be downloaded, also in the browser.

Thanks!
Ioan

oo_processing · by **oo_processing** » Wed Jan 15, 2020 4:25 pm America/New_York

I am unable to replicate the wget as it hangs. I tried in two places.

[bmurch@optics0 cron]$ wget oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1ATTNR.P2020014.1800.003
--2020-01-15 16:22:01-- http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1ATTNR.P2020014.1800.003
Resolving oceandata.sci.gsfc.nasa.gov... 2001:4d0:2418:128::84, xx.xxx.xx.xx
Connecting to oceandata.sci.gsfc.nasa.gov|2001:4d0:2418:128::84|:80... failed: Connection timed out.
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:80...

And:
[bmurch@dell8 ~]$ wget oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1ATTNR.P2020014.1800.003
--2020-01-15 16:23:28-- http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1ATTNR.P2020014.1800.003
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:80...

oo_processing · by **oo_processing** » Wed Jan 15, 2020 4:36 pm America/New_York

It also fails on the command line (the previous example was in a scripting situation) but with a 503 error opposed to the 302:
[bmurch@optics0 test]$ getanc.py /cms_zfs/sat_data/viirs/l1a/2020/015/V2020015122400.L1A_SNPP.nc --ancdb=./ancillary_data.db
VIIRS
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 503 - Service Temporarily Unavailable
*** ERROR: The HTTP transfer failed with status code 503.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/N202001512_MET_NCEP_6h.hdf
***
*** Also check to make sure you have write permissions under the directory:
*** /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/015
()
[66261 refs]

gnwiii · by **gnwiii** » Thu Jan 16, 2020 7:07 am America/New_York

Using Fedora 31 with python3, get_anc.py is failing for me. I can however download the "remote file" URL in the error message using either wget or curl with the credentials in ~/.netrc and options suggested in Download Methods. I don't have time now to try Ubuntu or MacOS -- it would be useful to know which (if any) configuration does work.

bruce · by **bruce** » Thu Jan 16, 2020 8:51 am America/New_York

Did I miss a step somewhere? I can download L1A files using wget and all of the new rigamarole but

bcb@modis:/v/aqua_data/gom$ modis_GEO.py --refreshDB --verbose A2020013173000.L1A_LAC
Determining required attitude and ephemeris files...
Searching database: /seadas/seadas-7.5.3/ocssw/var/ancillary_data.db
()
Input file: A2020013173000.L1A_LAC
Sensor : modisa
Start time: 2020013173000
End time : 2020013173500
()
Downloading 'PM1EPHND.P2020013.1200.003' to /seadas/seadas-7.5.3/ocssw/var/anc/2020/013
Connection interrupted, retrying up to 5 more time(s)
Connection error, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
URL attempted: oceandata.sci.gsfc.nasa.gov
Well, this is embarrassing...an error occurred that we just cannot get past...
Here is what we know: ('The read operation timed out',)
Please retry this request at a later time.
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2020013.1200.003
***
*** Also check to make sure you have write permissions under the directory:
*** /seadas/seadas-7.5.3/ocssw/var/anc/2020/013
()
Cannot create geolocation from A2020013173000.L1A_LAC; exiting.

What magic incantation do I need to say to get the "other" bits to work?

gnwiii · by **gnwiii** » Thu Jan 16, 2020 9:06 am America/New_York

These failures appear to be with get_anc.py (update_luts.py "works for me"). It may be useful to report your OS details and python version in case this is a distro issue. Just for completeness you might want to try running get_anc.py directly and also wget and curl downloads of using https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2020013.1200.003 (from the error message in your post).