wget redirection

by **OB ODPS - jgwilding** » Thu May 21, 2020 9:47 pm America/New_York

There was supposed to be network maintenance going on between 17:00-00:00 EDT that could cause intermittent connectivity problems.

john

emaure · by **emaure** » Thu May 21, 2020 10:53 pm America/New_York

Just an additional information

Trying direct download through the browser (chrome), redirection return error:

 .:. ERROR .:.
OceanColor Biology Processing Group (OBPG)
Sorry, an error has occurred. Use the back button to return to the previous page or go to the Ocean Color Home Page.

Maybe the maintenance explains the errors but if not, I am adding the above hoping that it will be useful.

oo_processing · by **oo_processing** » Fri May 22, 2020 9:10 am America/New_York

Note that the problem exists for anc data as well
[bmurch@optics0 ~]$ ll /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0???.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0200.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0400.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0600.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0800.003

/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0200.003:    HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0400.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0600.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0800.003:    HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1000.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1200.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1400.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1600.003: data
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1800.003:    HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.2000.003: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.2200.003: HTML document text

OB.DAACx - SeanBailey · by **OB.DAACx - SeanBailey** » Fri May 22, 2020 9:36 am America/New_York

I don't have confirmation that the issue was related to the maintenance done on the authentication servers...but the problem did seem to start about that time...
A minor change to how we connect to them has been put in place and seems to have fixed the issue...wondering if it fixes the previous max redirects error as well.

Try your downloads again and let me know if it's still an issue for you.

Sean

bruce · by **bruce** » Fri May 22, 2020 9:46 am America/New_York

Well I just tried my automated download and I still get redirection problems...

B

OB.DAACx - SeanBailey · by **OB.DAACx - SeanBailey** » Fri May 22, 2020 9:47 am America/New_York

...but not a 403 error?

bruce · by **bruce** » Fri May 22, 2020 10:16 am America/New_York

Just the redirection error (but I never got a 403 error, so from my perspective, no change)

OB.DAACx - SeanBailey · by **OB.DAACx - SeanBailey** » Fri May 22, 2020 10:19 am America/New_York

Well, at least the situation didn't get worse :wink:
Guess we still have some forensics to do...

alaroy · by **alaroy** » Fri May 22, 2020 11:34 am America/New_York

For whatever it's worth our downloads seem to be back to normal starting about 13:00 UTC (aka 9 EDT) today

oo_processing · by **oo_processing** » Fri May 22, 2020 11:46 am America/New_York

I have the same problem with redirections.

< HTTP/1.1 302 Found
< Server: nginx
< Date: Fri, 22 May 2020 15:25:33 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=60
< Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict
< Expires: Mon, 01 Jan 1970 00:00:00 GMT
< Cache-Control: no-cache, must-revalidate, max-age=0, no-store
< Pragma: no-cache
* Replaced cookie app-obdaac="880e0858d5c227ef26f66a3c8c0344c38f0c9875" for domain oceandata.sci.gsfc.nasa.gov, path /, expire 0
< Set-Cookie: app-obdaac=880e0858d5c227ef26f66a3c8c0344c38f0c9875; path=/; secure
< Referrer-Policy: no-referrer
< Expect-CT: max-age=31536000, enforce
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Content-Security-Policy: upgrade-insecure-requests; default-src 'self' oceancolor.gsfc.nasa.gov data:; script-src 'self' 'unsafe-inline' 'unsafe-eval' www.google-analytics.com www.googletagmanager.com cdn.earthdata.nasa.gov dap.digitalgov.gov data:; style-src 'self' 'unsafe-inline' code.jquery.com cdn.earthdata.nasa.gov; img-src 'self' data: oceancolor.gsfc.nasa.gov www.google-analytics.com cdn.earthdata.nasa.gov
<
* Ignoring the response-body
{ [data not shown]
0    0 0    0 0    0 0 0 --:--:-- 0:00:06 --:--:--    0* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact
* Issue another request to this URL: 'https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict'
* Re-using existing connection! (#1) with host urs.earthdata.nasa.gov
* Connected to urs.earthdata.nasa.gov (2001:4d0:241a:4081::89) port 443 (#1)
* Server auth using Basic with user 'oo_processing'

> GET /oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict HTTP/1.1
> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.11 libidn/1.18 libssh2/1.4.2
> Host: urs.earthdata.nasa.gov
> Accept: */*
> Cookie: _urs-gui_session=470e1acb2262074a485b8519e88c629a; urs_user_already_logged=yes
>

< HTTP/1.1 302 Found
< Server: nginx/1.17.5
< Date: Fri, 22 May 2020 15:25:33 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin-when-cross-origin
< Access-Control-Allow-Origin: null
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET, POST
< Access-Control-Expose-Headers: true
< Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=37602330dd13da46df4ff6d4fd0dff6aff91e3694c68e428c649a2b9d6ee5173
< Cache-Control: no-cache
* Replaced cookie urs_user_already_logged="yes" for domain earthdata.nasa.gov, path /, expire 1590247533
< Set-Cookie: urs_user_already_logged=yes; domain=earthdata.nasa.gov; path=/; expires=Sat, 23 May 2020 15:25:33 GMT
* Replaced cookie _urs-gui_session="470e1acb2262074a485b8519e88c629a" for domain urs.earthdata.nasa.gov, path /, expire 1590247533
< Set-Cookie: _urs-gui_session=470e1acb2262074a485b8519e88c629a; path=/; expires=Sat, 23 May 2020 15:25:33 GMT; HttpOnly
< X-Request-Id: 8efb344a-907d-4803-9410-70715f4df3ff
< X-Runtime: 0.098615
< Strict-Transport-Security: max-age=31536000
<
* Ignoring the response-body
{ [data not shown]
191 191 0 191 0 0 28 0 --:--:-- 0:00:06 --:--:-- 28* Connection #1 to host urs.earthdata.nasa.gov left intact
* Maximum (50) redirects followed

curl: (47) Maximum (50) redirects followed
* Closing connection #0
* Closing connection #1
$curl_command_returns: