wget redirection

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 139
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0
Been thanked: 1 time

wget redirection

by OB ODPS - jgwilding » Thu May 21, 2020 9:47 pm America/New_York

There was supposed to be network maintenance going on between 17:00-00:00 EDT that could cause intermittent connectivity problems.

john

Tags:

emaure
Posts: 27
Joined: Tue Mar 26, 2013 7:18 am America/New_York
Answers: 0

wget redirection

by emaure » Thu May 21, 2020 10:53 pm America/New_York

Just an additional information

Trying direct download through the browser (chrome), redirection return error:
              .:. ERROR .:.
OceanColor Biology Processing Group (OBPG)
Sorry, an error has occurred. Use the back button to return to the previous page or go to the Ocean Color Home Page.


Maybe the maintenance explains the errors but if not, I am adding the above hoping that it will be useful.

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

wget redirection

by oo_processing » Fri May 22, 2020 9:10 am America/New_York

Note that the problem exists for anc data as well
[bmurch@optics0 ~]$ ll /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0???.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0200.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0400.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0600.003
-rw-rw-r-- 1 bmurch cms_optics 665 May 22 00:13 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0800.003

/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0200.003:     HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0400.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0600.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.0800.003:     HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1000.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1200.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1400.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1600.003:      data                                               
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.1800.003:     HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.2000.003:      HTML document text                                 
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/138/PM1ATTNR.P2020138.2200.003:      HTML document text

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

wget redirection

by OB.DAAC - SeanBailey » Fri May 22, 2020 9:36 am America/New_York

I don't have confirmation that the issue was related to the maintenance done on the authentication servers...but the problem did seem to start about that time...
A minor change to how we connect to them has been put in place and seems to have fixed the issue...wondering if it fixes the previous max redirects error as well.

Try your downloads again and let me know if it's still an issue for you.

Sean

bruce
Posts: 85
Joined: Thu Mar 17, 2005 4:36 pm America/New_York
Answers: 0

wget redirection

by bruce » Fri May 22, 2020 9:46 am America/New_York

Well I just tried my automated download and I still get redirection problems...

B

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

wget redirection

by OB.DAAC - SeanBailey » Fri May 22, 2020 9:47 am America/New_York

...but not a 403 error?

bruce
Posts: 85
Joined: Thu Mar 17, 2005 4:36 pm America/New_York
Answers: 0

wget redirection

by bruce » Fri May 22, 2020 10:16 am America/New_York

Just the redirection error (but I never got a 403 error, so from my perspective, no change)

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

wget redirection

by OB.DAAC - SeanBailey » Fri May 22, 2020 10:19 am America/New_York

Well,  at least the situation didn't get worse :wink:
Guess we still have some forensics to do...

alaroy
Posts: 78
Joined: Tue Nov 21, 2006 3:33 pm America/New_York
Answers: 0

wget redirection

by alaroy » Fri May 22, 2020 11:34 am America/New_York

For whatever it's worth our downloads seem to be back to normal starting about 13:00 UTC (aka 9 EDT) today

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

wget redirection

by oo_processing » Fri May 22, 2020 11:46 am America/New_York

I have the same problem with redirections.

< HTTP/1.1 302 Found                                                                                                                                                              
< Server: nginx                                                                                                                                                                   
< Date: Fri, 22 May 2020 15:25:33 GMT                                                                                                                                             
< Transfer-Encoding: chunked                                                                                                                                                      
< Connection: keep-alive                                                                                                                                                          
< Keep-Alive: timeout=60                                                                                                                                                          
< Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict                                                                                                                                                                      
< Expires: Mon, 01 Jan 1970 00:00:00 GMT                                                                                                                                          
< Cache-Control: no-cache, must-revalidate, max-age=0, no-store                                                                                                                   
< Pragma: no-cache                                                                                                                                                                
* Replaced cookie app-obdaac="880e0858d5c227ef26f66a3c8c0344c38f0c9875" for domain oceandata.sci.gsfc.nasa.gov, path /, expire 0                                                  
< Set-Cookie: app-obdaac=880e0858d5c227ef26f66a3c8c0344c38f0c9875; path=/; secure                                                                                                 
< Referrer-Policy: no-referrer                                                                                                                                                    
< Expect-CT: max-age=31536000, enforce                                                                                                                                            
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload                                                                                                         
< Content-Security-Policy: upgrade-insecure-requests; default-src 'self' oceancolor.gsfc.nasa.gov data:; script-src 'self' 'unsafe-inline' 'unsafe-eval' www.google-analytics.com www.googletagmanager.com cdn.earthdata.nasa.gov dap.digitalgov.gov data:; style-src 'self' 'unsafe-inline' code.jquery.com cdn.earthdata.nasa.gov; img-src 'self' data: oceancolor.gsfc.nasa.gov www.google-analytics.com cdn.earthdata.nasa.gov                                                                                                                      
<                                                                                                                                                                                 
* Ignoring the response-body                                                                                                                                                      
{ [data not shown]                                                                                                                                                                
  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact                                     
* Issue another request to this URL: 'https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict'                                                                                                                                           
* Re-using existing connection! (#1) with host urs.earthdata.nasa.gov
* Connected to urs.earthdata.nasa.gov (2001:4d0:241a:4081::89) port 443 (#1)
* Server auth using Basic with user 'oo_processing'

> GET /oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict HTTP/1.1
> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.11 libidn/1.18 libssh2/1.4.2
> Host: urs.earthdata.nasa.gov
> Accept: */*
> Cookie: _urs-gui_session=470e1acb2262074a485b8519e88c629a; urs_user_already_logged=yes
>


< HTTP/1.1 302 Found
< Server: nginx/1.17.5
< Date: Fri, 22 May 2020 15:25:33 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin-when-cross-origin
< Access-Control-Allow-Origin: null
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET, POST
< Access-Control-Expose-Headers: true
< Location: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict?code=37602330dd13da46df4ff6d4fd0dff6aff91e3694c68e428c649a2b9d6ee5173
< Cache-Control: no-cache
* Replaced cookie urs_user_already_logged="yes" for domain earthdata.nasa.gov, path /, expire 1590247533
< Set-Cookie: urs_user_already_logged=yes; domain=earthdata.nasa.gov; path=/; expires=Sat, 23 May 2020 15:25:33 GMT
* Replaced cookie _urs-gui_session="470e1acb2262074a485b8519e88c629a" for domain urs.earthdata.nasa.gov, path /, expire 1590247533
< Set-Cookie: _urs-gui_session=470e1acb2262074a485b8519e88c629a; path=/; expires=Sat, 23 May 2020 15:25:33 GMT; HttpOnly
< X-Request-Id: 8efb344a-907d-4803-9410-70715f4df3ff
< X-Runtime: 0.098615
< Strict-Transport-Security: max-age=31536000
<
* Ignoring the response-body
{ [data not shown]
191   191    0   191    0     0     28      0 --:--:--  0:00:06 --:--:--    28* Connection #1 to host urs.earthdata.nasa.gov left intact
* Maximum (50) redirects followed

curl: (47) Maximum (50) redirects followed
* Closing connection #0
* Closing connection #1
$curl_command_returns:

Post Reply