Ruling out assymetric route (first)?

haag · by **haag** » Fri Jan 20, 2017 5:11 pm America/New_York

Hello,

We're having several issues with data downloading, and now, running SeaDAS. I tried patching for the https switchover, but am now (see below) just installing fresh to reduce "variables" in the debugging.

First I'd like to confirm it's not a asymmetric route issue. (I began debugging problems in November and some routing issues were seemingly corrected then, but problems quickly returned, and have remained intermittent.)

For example, right now (Fri Jan 20 16:06:13 CST 2017), I am trying a fresh manual install of SeaDAS and am seeing install_ocssw.py hang on different "parts" of the downloading. A rerun might get farther or not as far as just previously. Right now it seemed to be forging ahead, but then "timed out before SSL handshake". Subsequent runs now seem to hang earlier ("Installing common (2 of 13)").

We have been plagued by data downloads that work one minute and then fail on another attempt for another file. We've trying from various machine in subnets 130.39.188.x thru 130.39.191.x with similar behaviors.

So, to first eliminate the routing issue...

This machine is at xx.xxx.xx.xx, running a fully-updated CentOS 6.8, (I also rebuilt "git" to make sure it was linking with a "latest" OpenSSL. )

Here is a traceroute:

$ traceroute oceandata.sci.gsfc.nasa.gov
traceroute to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx), 30 hops max, 60 byte packets
 1  howe-e241a-4006-dsw-1.net.lsu.edu (xx.xxx.xx.xx)  0.439 ms  0.543 ms  0.617 ms
 2  mfcacsw-howedsw.cnet3.lsu.edu (xx.xxx.xx.xx)  0.294 ms  0.286 ms  0.319 ms
 3  csc.edge.frwl-mfca.csw-cnet1.lsu.edu (xx.xxx.xx.xx)  0.621 ms  0.577 ms  0.579 ms
 4  csc.bdr-csc.edge.frwl-cnet1.lsu.edu (xx.xxx.xx.xx)  1.574 ms  1.511 ms  1.514 ms
 5  ip-209-124-254-21.static.eatel.net (xx.xxx.xx.xx)  2.502 ms LONI-1418-LSUF-LONI.loni.org (xx.xxx.xx.xx)  0.991 ms ip-209-124-254-21.eatel.net (xx.xxx.xx.xx)  2.511 ms
 6  * * *
 7  rtr.houh.net.internet2.edu-et-10-2-0.loni.org (xx.xxx.xx.xx)  6.814 ms  6.814 ms  6.936 ms
 8  et-3-3-0.4070.rtsw.atla.net.internet2.edu (xx.xxx.xx.xx)  30.850 ms ae-1-3501.ear2.Washington1.Level3.net (xx.xxx.xx.xx)  2269.774 ms et-3-3-0.4070.rtsw.atla.net.internet2.edu (xx.xxx.xx.xx)  30.827 ms
 9  et-9-0-0.124.rtr.ashb.net.internet2.edu (xx.xxx.xx.xx)  44.125 ms SIMS-INC.ear2.Washington1.Level3.net (xx.xxx.xx.xx)  38.997 ms  42.696 ms
10  et-11-3-0-1275.clpk-core.maxgigapop.net (xx.xxx.xx.xx)  45.982 ms  45.992 ms  45.217 ms
11  xx.xxx.xx.xx (xx.xxx.xx.xx)  39.455 ms xx.xxx.xx.xx (xx.xxx.xx.xx)  46.693 ms  46.629 ms
12  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  45.971 ms  46.172 ms  46.056 ms
13  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  43.364 ms  43.515 ms *
14  * * *
15  * * *
16  * * *

And to oceancolor:


$ traceroute oceancolor.gsfc.nasa.gov
traceroute to oceancolor.gsfc.nasa.gov (xx.xxx.xx.xx), 30 hops max, 60 byte packets
 1  howe-e241a-4006-dsw-1.net.lsu.edu (xx.xxx.xx.xx)  0.533 ms  0.629 ms  0.676 ms
 2  mfcacsw-howedsw.cnet3.lsu.edu (xx.xxx.xx.xx)  0.326 ms  0.309 ms  0.290 ms
 3  csc.edge.frwl-mfca.csw-cnet1.lsu.edu (xx.xxx.xx.xx)  0.589 ms  0.623 ms  0.605 ms
 4  csc-118-k7-6509-bdr.net.lsu.edu (xx.xxx.xx.xx)  1.434 ms  1.461 ms  1.432 ms
 5  LSUD-1424-LSUF-LONI.loni.org (xx.xxx.xx.xx)  0.961 ms ip-209-124-254-21.static.eatel.net (xx.xxx.xx.xx)  2.239 ms LSUD-1424-LSUF-LONI.loni.org (xx.xxx.xx.xx)  0.933 ms
 6  * * *
 7  * * rtr.houh.net.internet2.edu-et-10-2-0.loni.org (xx.xxx.xx.xx)  6.920 ms
 8  et-3-3-0.4070.rtsw.atla.net.internet2.edu (xx.xxx.xx.xx)  30.860 ms * *
 9  et-9-0-0.124.rtr.ashb.net.internet2.edu (xx.xxx.xx.xx)  44.012 ms  44.110 ms  44.053 ms
10  et-11-3-0-1275.clpk-core.maxgigapop.net (xx.xxx.xx.xx)  45.847 ms xx.xxx.xx.xx (xx.xxx.xx.xx)  39.810 ms  43.523 ms
11  xx.xxx.xx.xx (xx.xxx.xx.xx)  43.557 ms  43.083 ms  39.586 ms
12  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  46.821 ms  46.150 ms  46.026 ms
13  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  39.843 ms  43.244 ms *
14  * * *
15  * * *
16  * * *

I'm grateful for any and all assistance!
Alaric Haag

gnwiii · by **gnwiii** » Sat Jan 21, 2017 7:15 am America/New_York

It may be helpful to provide tcptraceroute results, but this usually needs "sudo". Use port 443 for https, as in:

$ sudo tcptraceroute oceandata.sci.gsfc.nasa.gov 443
Selected device wlan0, address xx.xxx.xx.xx, port 26005 for outgoing packets
Tracing the path to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) on TCP port 443 (https), 30 hops max
 1  xx.xxx.xx.xx  51.261 ms  90.327 ms  96.741 ms
 2  * * *
[...]
18  * * *
19  gs6142obpg1020.sci.gsfc.nasa.gov (xx.xxx.xx.xx) [open]  1320.542 ms  1365.730 ms  1318.475 ms

haag · by **haag** » Mon Jan 23, 2017 10:40 am America/New_York

Many thanks!

Here are some results.


# tcptraceroute oceandata.sci.gsfc.nasa.gov 443
Selected device br0, address xx.xxx.xx.xx, port 58253 for outgoing packets
Tracing the path to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) on TCP port 443 (https), 30 hops max
 1  howe-e241a-4006-dsw-1.net.lsu.edu (xx.xxx.xx.xx)  0.466 ms  0.375 ms  0.305 ms
 2  xx.xxx.xx.xx  0.284 ms  0.753 ms  0.351 ms
 3  xx.xxx.xx.xx  0.629 ms  0.553 ms  0.603 ms
 4  csc.bdr-csc.edge.frwl-cnet1.lsu.edu (xx.xxx.xx.xx)  1.426 ms  0.890 ms  0.835 ms
 5  LSUD-1424-LSUF-LONI.loni.org (xx.xxx.xx.xx)  0.598 ms  0.528 ms  0.538 ms
 6  * * *
 7  rtr.houh.net.internet2.edu-et-10-2-0.loni.org (xx.xxx.xx.xx)  6.661 ms  6.481 ms  6.456 ms
 8  et-3-3-0.4070.rtsw.atla.net.internet2.edu (xx.xxx.xx.xx)  30.415 ms  30.404 ms  30.391 ms
 9  et-9-0-0.124.rtr.ashb.net.internet2.edu (xx.xxx.xx.xx)  43.784 ms  43.669 ms  43.687 ms
10  et-11-3-0-1275.clpk-core.maxgigapop.net (xx.xxx.xx.xx)  45.447 ms  45.421 ms  45.437 ms
11  xx.xxx.xx.xx  52.982 ms  46.239 ms  54.093 ms
12  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  46.784 ms  48.051 ms  46.392 ms
13  gs6142obpg1001.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  45.488 ms  45.499 ms  45.499 ms
14  gs6142obpg1020.sci.gsfc.nasa.gov (xx.xxx.xx.xx) [open]  45.598 ms  45.612 ms  45.606 ms

In case there's any concern about me using a bridge (br0) interface, here are results from another machine in our lab that isn't.


# tcptraceroute oceandata.sci.gsfc.nasa.gov 443
Selected device eth0, address xx.xxx.xx.xx, port 48256 for outgoing packets
Tracing the path to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) on TCP port 443 (https), 30 hops max
 1  howe-e241a-4006-dsw-1.net.lsu.edu (xx.xxx.xx.xx)  0.621 ms  0.438 ms  0.389 ms
 2  xx.xxx.xx.xx  0.405 ms  0.329 ms  0.381 ms
 3  xx.xxx.xx.xx  0.656 ms  0.569 ms  0.586 ms
 4  csc.bdr-csc.edge.frwl-cnet1.lsu.edu (xx.xxx.xx.xx)  1.266 ms  0.819 ms  0.849 ms
 5  LSUD-1424-LSUF-LONI.loni.org (xx.xxx.xx.xx)  0.522 ms  0.521 ms  0.511 ms
 6  * * *
 7  rtr.houh.net.internet2.edu-et-10-2-0.loni.org (xx.xxx.xx.xx)  6.538 ms  6.475 ms  6.466 ms
 8  et-3-3-0.4070.rtsw.atla.net.internet2.edu (xx.xxx.xx.xx)  30.434 ms  30.429 ms  30.483 ms
 9  et-9-0-0.124.rtr.ashb.net.internet2.edu (xx.xxx.xx.xx)  43.643 ms  43.642 ms  43.618 ms
10  et-11-3-0-1275.clpk-core.maxgigapop.net (xx.xxx.xx.xx)  45.495 ms  45.479 ms  45.520 ms
11  xx.xxx.xx.xx  45.664 ms  50.875 ms  53.830 ms
12  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  45.896 ms  45.731 ms  47.847 ms
13  gs6142obpg1001.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  46.211 ms  46.207 ms  46.484 ms
14  gs6142obpg1020.sci.gsfc.nasa.gov (xx.xxx.xx.xx) [open]  46.332 ms  46.320 ms  46.255 ms

And same pair for oceancolor site:


# tcptraceroute oceancolor.gsfc.nasa.gov 443
Selected device br0, address xx.xxx.xx.xx, port 39056 for outgoing packets
Tracing the path to oceancolor.gsfc.nasa.gov (xx.xxx.xx.xx) on TCP port 443 (https), 30 hops max
 1  howe-e241a-4006-dsw-1.net.lsu.edu (xx.xxx.xx.xx)  0.501 ms  0.358 ms  0.311 ms
 2  xx.xxx.xx.xx  0.305 ms  0.310 ms  0.266 ms
 3  xx.xxx.xx.xx  0.641 ms  0.620 ms  0.581 ms
 4  csc-118-k7-6509-bdr.net.lsu.edu (xx.xxx.xx.xx)  1.226 ms  0.785 ms  0.855 ms
 5  ip-209-124-254-21.static.eatel.net (xx.xxx.xx.xx)  1.904 ms  1.621 ms  1.543 ms
 6  * * *
 7  * * *
 8  * * *
 9  SIMS-INC.ear2.Washington1.Level3.net (xx.xxx.xx.xx)  130.804 ms  151.025 ms  56.359 ms
10  xx.xxx.xx.xx  49.665 ms  38.764 ms  38.757 ms
11  xx.xxx.xx.xx  39.002 ms  38.957 ms  38.878 ms
12  xx.xxx.xx.xx  42.218 ms  51.863 ms  39.056 ms
13  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  39.297 ms  41.845 ms  39.436 ms
14  gs6142obpg1001.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  39.822 ms  39.776 ms  39.736 ms
15  oceancolor.sci.gsfc.nasa.gov (xx.xxx.xx.xx) [open]  39.164 ms  40.758 ms  39.499 ms

and from "eth0" server:


# tcptraceroute oceancolor.gsfc.nasa.gov 443
Selected device eth0, address xx.xxx.xx.xx, port 44578 for outgoing packets
Tracing the path to oceancolor.gsfc.nasa.gov (xx.xxx.xx.xx) on TCP port 443 (https), 30 hops max
 1  howe-e241a-4006-dsw-1.net.lsu.edu (xx.xxx.xx.xx)  0.640 ms  0.488 ms  0.411 ms
 2  xx.xxx.xx.xx  200.765 ms  11.540 ms  3.441 ms
 3  xx.xxx.xx.xx  0.707 ms  0.575 ms  0.635 ms
 4  csc-118-k7-6509-bdr.net.lsu.edu (xx.xxx.xx.xx)  1.401 ms  0.890 ms  0.846 ms
 5  ip-209-124-254-21.eatel.net (xx.xxx.xx.xx)  1.698 ms  1.620 ms  1.489 ms
 6  * * *
 7  * * *
 8  * * *
 9  SIMS-INC.ear2.Washington1.Level3.net (xx.xxx.xx.xx)  42.312 ms  42.289 ms  42.319 ms
10  xx.xxx.xx.xx  42.525 ms  42.534 ms  42.596 ms
11  xx.xxx.xx.xx  43.427 ms  43.378 ms  43.233 ms
12  xx.xxx.xx.xx  51.765 ms  48.941 ms  53.867 ms
13  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  43.595 ms  43.399 ms  43.055 ms
14  gs6142obpg1001.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  43.405 ms  43.443 ms  43.411 ms
15  oceancolor.sci.gsfc.nasa.gov (xx.xxx.xx.xx) [open]  43.532 ms  43.529 ms  43.379 ms

Again, I'm most grateful for the assistance!
Alaric

by **OB.DAAC - SeanBailey** » Mon Jan 23, 2017 4:27 pm America/New_York

Alaric,

From the information you've provided we can rule out the assymetric route problem :grin:
There was a modification to the firewall on 20 Jan 2017 that may improve things for you.
Are you still experiencing difficulties?

Sean

haag · by **haag** » Mon Jan 23, 2017 4:59 pm America/New_York

Many thanks Sean!

I'm not sure I'm "out of the woods" yet.

For example, using "curl" works, then doesn't, as shown below from just a few minutes ago.

I'm running curl version:


curl --version
curl 7.49.0 (x86_64-pc-linux-gnu) libcurl/7.49.0 OpenSSL/1.0.2j zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

Here is the recent interaction.


$ curl -v -O https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2017009071200.L2_SNPP_SST.nc
*   Trying xx.xxx.xx.xx...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /software/anaconda2/ssl/cacert.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [109 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3184 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [180 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=oceancolor.sci.gsfc.nasa.gov
*  start date: Dec 14 00:00:00 2016 GMT
*  expire date: Dec 14 23:59:59 2019 GMT
*  subjectAltName: host "oceandata.sci.gsfc.nasa.gov" matched cert's "oceandata.sci.gsfc.nasa.gov"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.
} [5 bytes data]

> GET /cgi/getfile/V2017009071200.L2_SNPP_SST.nc HTTP/1.1
> Host: oceandata.sci.gsfc.nasa.gov
> User-Agent: curl/7.49.0
> Accept: */*
>

{ [5 bytes data]
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0< HTTP/1.1 200 OK
< Server: nginx
< Date: Mon, 23 Jan 2017 21:45:43 GMT
< Content-Type: application/octet-stream
< Content-Length: 86066935
< Connection: keep-alive
< Last-Modified: Mon, 09 Jan 2017 13:50:52 GMT
< Content-Disposition: attachment; filename=V2017009071200.L2_SNPP_SST.nc
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<
{ [16031 bytes data]
100 82.0M 100 82.0M 0 0 16.3M 0 0:00:05 0:00:05 --:--:-- 21.1M
* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact

I waited, albeit briefly, but not an instant retry, and got:


$ curl -v -O https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2017009071200.L2_SNPP_SST.nc
*   Trying xx.xxx.xx.xx...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /software/anaconda2/ssl/cacert.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:03:39 --:--:--     0

Kind regards,
Alaric

Alaric S. Haag, Systems Administrator Earth Scan Laboratory
Louisiana State University http://www.esl.lsu.edu/
Coastal Studies Institute Howe-Russell Bldg. Rm 331
Office: 225 578 6438 Facsimile: 225 578 2520
Email: haag â€œatâ€ lsu â€œdotâ€ edu

haag · by **haag** » Mon Jan 23, 2017 5:03 pm America/New_York

And just as an added note, both today, and when I posted Friday, I composed the message and clicked "Post" to have the browser report it couldn't connect to the server. A click "back" and second click of "Post" worked...

Don't know if that's significant, but adds more to the mystery...

Thanks again!

Kind regards,
Alaric

Alaric S. Haag, Systems Administrator Earth Scan Laboratory
Louisiana State University http://www.esl.lsu.edu/
Coastal Studies Institute Howe-Russell Bldg. Rm 331
Office: 225 578 6438 Facsimile: 225 578 2520
Email: haag “at” lsu “dot” edu

by **OB.DAAC - SeanBailey** » Wed Jan 25, 2017 4:24 pm America/New_York

Alaric,

Are you still experiencing difficulties? I'm seeing the occasional "SSL handshaking" error in our logs with an IP I think is yours
(there are two under the subnet 130.39.189/ reporting the errors) But oddly there are also successful data transfers, so
there seems to be some weirdness on your end that sometimes fails the SSL handshake and other times is successful.

Sean

haag · by **haag** » Wed Jan 25, 2017 6:26 pm America/New_York

Sean,

Many thanks! I am seeing inconsistent behavior, and I was trying to rule things out before posting.

I saw some prior comments about using Python 2.7 and I knew my installation was aging, so I setup a completely updated anaconda2 installation, but a simple test program that uses "urllib2.urlopen" will succeed, and then hang with a handshake error... then it might work again, or hang again...

And so does "curl", which looks like:


$ curl -v -O https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2017010065400.L2_SNPP_SST.nc
*   Trying xx.xxx.xx.xx...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /software/anaconda2/ssl/cacert.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [109 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3184 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [179 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=oceancolor.sci.gsfc.nasa.gov
*  start date: Dec 14 00:00:00 2016 GMT
*  expire date: Dec 14 23:59:59 2019 GMT
*  subjectAltName: host "oceandata.sci.gsfc.nasa.gov" matched cert's "oceandata.sci.gsfc.nasa.gov"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.
} [5 bytes data]

> GET /cgi/getfile/V2017010065400.L2_SNPP_SST.nc HTTP/1.1
> Host: oceandata.sci.gsfc.nasa.gov
> User-Agent: curl/7.49.0
> Accept: */*
>

{ [5 bytes data]
< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 25 Jan 2017 23:21:32 GMT
< Content-Type: application/octet-stream
< Content-Length: 89544038
< Connection: keep-alive
< Last-Modified: Wed, 25 Jan 2017 22:26:36 GMT
< Content-Disposition: attachment; filename=V2017010065400.L2_SNPP_SST.nc
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<
{ [16031 bytes data]
100 85.3M 100 85.3M 0 0 21.2M 0 0:00:04 0:00:04 --:--:-- 21.2M
* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact

when it succeeds, and when it hangs, looks like:


$ curl -v -O https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2017010065400.L2_SNPP_SST.nc
*   Trying xx.xxx.xx.xx...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /software/anaconda2/ssl/cacert.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0

I'm completely confused by this behavior...

Alaric

haag · by **haag** » Thu Jan 26, 2017 11:00 am America/New_York

I've been trying various things to try to use a "latest" version of OpenSSL, but even CentOS 7 is using 1.0.1e.

Anyhow, in case it's useful, this is the report from the openssl that comes with Anaconda:


$ openssl version -a
OpenSSL 1.0.2j  26 Sep 2016
built on: reproducible build, date unspecified
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -m64 -DL_ENDIAN -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
OPENSSLDIR: "/software/anaconda2/ssl"

The CentOS 6 distribution version (1.0.1e-fips) adds "md2(int)" to the list of encryption options, and I am seeing the same "intermittent" behavior with it...

And just tested "wget" with similar results; works, works, hangs...

Hopefully something helpful in there!

Many thanks for any attention and suggestions!

Alaric S. Haag, Systems Administrator Earth Scan Laboratory
Louisiana State University http://www.esl.lsu.edu/
Coastal Studies Institute Howe-Russell Bldg. Rm 331
Office: 225 578 6438 Facsimile: 225 578 2520
Email: haag “at” lsu “dot” edu

gnwiii · by **gnwiii** » Thu Jan 26, 2017 2:35 pm America/New_York

I have a CentOS7 system with OpenSSL 1.0.1e-fips 11 Feb 2013 which works intermittently, as do other systems here (Bedford Institute of Oceanography in Nova Scotia). I see intermittent "no route to host" errors and "host not responding in web browsers. Our IT people say we are getting a network upgrade "real soon now", so I'm waiting to see if that solves these problems.