Wget constantly redirected
Posted: Fri Feb 21, 2020 10:49 am America/New_York
Hi OC forum,
Quick question ... I'm using the new wget procedure and it works perfectly until..it doesn't. My script plugs along perfectly for a couple hours and then wget will suddenly be redirected and not find the files. (Example below) By deleting the .urs_cookies file this seems to fix it. Just wondering if it's possible to tell if this is an error on my end (eg network?) or why this is happening? I'm working on a large processing and it would be nice to be able to leave it without having to check back constantly to make sure it's still working.
Example:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2016201174500.L2_LAC_OC.nc
--2020-02-21 11:38:37-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2016201174500.L2_LAC_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2016201174500.L2_LAC_OC.nc [following]
--2020-02-21 11:38:37-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2016201174500.L2_LAC_OC.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code [following]
--2020-02-21 11:38:37-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=7c6cfe60597147c8d27c205eceed61a246f5cdbb37f2c8a36b7e7119d78e6289 [following]
--2020-02-21 11:38:38-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=7c6cfe60597147c8d27c205eceed61a246f5cdbb37f2c8a36b7e7119d78e6289
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code [following]
--2020-02-21 11:38:38-- https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=24d69431beb953c1b53e7e01f9a9695febf519af9e04f82a1527452d197645f2 [following]
--2020-02-21 11:38:38-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=24d69431beb953c1b53e7e01f9a9695febf519af9e04f82a1527452d197645f2
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code [following]
--2020-02-21 11:38:38-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=afb8a035a75c5424a47ebfa2438e4360acdfa8fead849db0e26822df61a6ec55 [following]
--2020-02-21 11:38:39-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=afb8a035a75c5424a47ebfa2438e4360acdfa8fead849db0e26822df61a6ec55
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code [following]
--2020-02-21 11:38:39-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=f3b283d52a8ae0845c67c59cd228a760e71fc2112609b666c4e7bb87154e9aae [following]
--2020-02-21 11:38:39-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=f3b283d52a8ae0845c67c59cd228a760e71fc2112609b666c4e7bb87154e9aae
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict [following]
--2020-02-21 11:38:40-- https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
----ETC...---
20 redirections exceeded.
Quick question ... I'm using the new wget procedure and it works perfectly until..it doesn't. My script plugs along perfectly for a couple hours and then wget will suddenly be redirected and not find the files. (Example below) By deleting the .urs_cookies file this seems to fix it. Just wondering if it's possible to tell if this is an error on my end (eg network?) or why this is happening? I'm working on a large processing and it would be nice to be able to leave it without having to check back constantly to make sure it's still working.
Example:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2016201174500.L2_LAC_OC.nc
--2020-02-21 11:38:37-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2016201174500.L2_LAC_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2016201174500.L2_LAC_OC.nc [following]
--2020-02-21 11:38:37-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2016201174500.L2_LAC_OC.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code [following]
--2020-02-21 11:38:37-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=7c6cfe60597147c8d27c205eceed61a246f5cdbb37f2c8a36b7e7119d78e6289 [following]
--2020-02-21 11:38:38-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=7c6cfe60597147c8d27c205eceed61a246f5cdbb37f2c8a36b7e7119d78e6289
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code [following]
--2020-02-21 11:38:38-- https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=24d69431beb953c1b53e7e01f9a9695febf519af9e04f82a1527452d197645f2 [following]
--2020-02-21 11:38:38-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=24d69431beb953c1b53e7e01f9a9695febf519af9e04f82a1527452d197645f2
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code [following]
--2020-02-21 11:38:38-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=afb8a035a75c5424a47ebfa2438e4360acdfa8fead849db0e26822df61a6ec55 [following]
--2020-02-21 11:38:39-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=afb8a035a75c5424a47ebfa2438e4360acdfa8fead849db0e26822df61a6ec55
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code [following]
--2020-02-21 11:38:39-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=f3b283d52a8ae0845c67c59cd228a760e71fc2112609b666c4e7bb87154e9aae [following]
--2020-02-21 11:38:39-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=f3b283d52a8ae0845c67c59cd228a760e71fc2112609b666c4e7bb87154e9aae
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict [following]
--2020-02-21 11:38:40-- https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
----ETC...---
20 redirections exceeded.