Ruling out assymetric route (first)?

[haag]

Some host outside my lab, but on the campus net have experienced the "hang" behavior.

I've been taking a look with tcpdump. These examples were taken around 10:29AM Central today.

When it works, the transmission looks like:


1   0.000000 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 74 46024 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1557369173 TSecr=0 WS=128
  2   0.046195 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 74 https > 46024 [SYN, ACK] Seq=0 Ack=1 Win=35792 Len=0 MSS=8960 SACK_PERM=1 TSval=189256809 TSecr=1557369173 WS=16384
  3   0.046251 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 46024 > https [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=1557369220 TSecr=189256809
  4   0.068015 xx.xxx.xx.xx -> xx.xxx.xx.xx SSL 583 Client Hello
  5   0.113760 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 https > 46024 [ACK] Seq=1 Ack=518 Win=49152 Len=0 TSval=189256826 TSecr=1557369242
  6   0.115989 xx.xxx.xx.xx -> xx.xxx.xx.xx TLSv1.2 3562 Server Hello, Certificate, Server Key Exchange, Server Hello Done
  7   0.116035 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 46024 > https [ACK] Seq=518 Ack=3497 Win=17536 Len=0 TSval=1557369290 TSecr=189256827
  8   0.126163 xx.xxx.xx.xx -> xx.xxx.xx.xx TLSv1.2 192 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
  9   0.172261 xx.xxx.xx.xx -> xx.xxx.xx.xx TLSv1.2 117 Change Cipher Spec, Encrypted Handshake Message
10   0.173043 xx.xxx.xx.xx -> xx.xxx.xx.xx TLSv1.2 227 Application Data
11   0.255486 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 https > 46024 [ACK] Seq=3548 Ack=805 Win=49152 Len=0 TSval=189256862 TSecr=1557369347
12   0.709263 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 7306 [TCP segment of a reassembled PDU]
13   0.709322 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 46024 > https [ACK] Seq=805 Ack=10788 Win=20480 Len=0 TSval=1557369883 TSecr=189256975
14   0.709351 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 7306 [TCP segment of a reassembled PDU]
15   0.709380 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 46024 > https [ACK] Seq=805 Ack=18028 Win=23296 Len=0 TSval=1557369883 TSecr=189256975
16   0.755368 xx.xxx.xx.xx -> xx.xxx.xx.xx TLSv1.2 8754 Application Data
17   0.755424 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 46024 > https [ACK] Seq=805 Ack=26716 Win=26240 Len=0 TSval=1557369929 TSecr=189256986
18   0.755454 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 5858 [TCP segment of a reassembled PDU]
19   0.755482 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 46024 > https [ACK] Seq=805 Ack=32508 Win=29184 Len=0 TSval=1557369929 TSecr=189256986
20   0.755488 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 1514 [TCP segment of a reassembled PDU]
21   0.755502 xx.xxx.xx.xx -> xx.xxx.xx.xx TLSv1.2 7306 Application Data
... file transfer stuff deleted...


when it hangs, I see:


  1   0.000000 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 74 46028 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1557385948 TSecr=0 WS=128
  2   0.043652 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 74 https > 46028 [SYN, ACK] Seq=0 Ack=1 Win=35792 Len=0 MSS=8960 SACK_PERM=1 TSval=189261002 TSecr=1557385948 WS=16384
  3   0.043709 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 46028 > https [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=1557385992 TSecr=189261002
  4   0.065362 xx.xxx.xx.xx -> xx.xxx.xx.xx SSL 583 Client Hello
  5   0.316203 xx.xxx.xx.xx -> xx.xxx.xx.xx SSL 583 [TCP Retransmission] Client Hello
  6   0.818214 xx.xxx.xx.xx -> xx.xxx.xx.xx SSL 583 [TCP Retransmission] Client Hello
  7   1.041081 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 74 [TCP Retransmission] https > 46028 [SYN, ACK] Seq=0 Ack=1 Win=35792 Len=0 MSS=8960 SACK_PERM=1 TSval=189261252 TSecr=1557385948 WS=16384
  8   1.041123 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 [TCP Dup ACK 6#1] 46028 > https [ACK] Seq=518 Ack=1 Win=14720 Len=0 TSval=1557386989 TSecr=189261002
  9   1.822234 xx.xxx.xx.xx -> xx.xxx.xx.xx SSL 583 [TCP Retransmission] Client Hello
10   3.040770 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 74 [TCP Retransmission] https > 46028 [SYN, ACK] Seq=0 Ack=1 Win=35792 Len=0 MSS=8960 SACK_PERM=1 TSval=189261752 TSecr=1557385948 WS=16384
11   3.040819 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 [TCP Dup ACK 9#1] 46028 > https [ACK] Seq=518 Ack=1 Win=14720 Len=0 TSval=1557388989 TSecr=189261002
12   3.830244 xx.xxx.xx.xx -> xx.xxx.xx.xx SSL 583 [TCP Retransmission] Client Hello
13   7.040788 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 74 [TCP Retransmission] https > 46028 [SYN, ACK] Seq=0 Ack=1 Win=35792 Len=0 MSS=8960 SACK_PERM=1 TSval=189262752 TSecr=1557385948 WS=16384
14   7.040835 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 [TCP Dup ACK 12#1] 46028 > https [ACK] Seq=518 Ack=1 Win=14720 Len=0 TSval=1557392989 TSecr=189261002
15   7.846224 xx.xxx.xx.xx -> xx.xxx.xx.xx SSL 583 [TCP Retransmission] Client Hello
16  15.070134 xx.xxx.xx.xx -> xx.xxx.xx.xx TCP 66 46028 > https [FIN, ACK] Seq=518 Ack=1 Win=14720 Len=0 TSval=1557401018 TSecr=189261002
17  15.878228 xx.xxx.xx.xx -> xx.xxx.xx.xx SSL 583 [TCP Retransmission] Client Hello


and nothing more.

It appears that the initial TCP connection gets acknowledged, but the HELO packet for the TLS negotiation does not. Firewall??

Sean, can your network guys shed any light on this? I can provide the raw pcap (tcpdump) files if needed.

I really appreciate any attention thrown at this!

Alaric

[OB.DAACx - SeanBailey]

Alaric,

Our network guru had this to say:

"My best guess from the tcpdump logs is packet loss somewhere in the path to us.
When his client sends the "Hello" our server does not seem to get the packet and
does not respond. He makes three(3) more "Client Hello" attempts and on the third
Hello we respond, then the client goes into a Hello/Syn ACK loop."

Anyway you can check for packet loss? 

Sean

[haag]

Sean,

Thanks! I revisiting routing with my guys, and they are requesting a traceroute from your end.

Can you please aim at xx.xxx.xx.xx, though it will die at a firewall before getting there.

Many thanks for all the assistance!

Alaric

[OB.DAACx - SeanBailey]

Here you go:
traceroute to xx.xxx.xx.xx (xx.xxx.xx.xx), 64 hops max, 40 byte packets
1  rtr-s28-40g.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  0.388 ms 0.233 ms  0.414 ms
2  rtr-hecn-sen-40g.sci.gsfc.nasa.gov (xx.xxx.xx.xx) 0.925 ms  9.950 ms  0.598 ms
3  xx.xxx.xx.xx (xx.xxx.xx.xx)  2.228 ms  1.200 ms  1.251 ms
4  ae0.mcln-core.maxgigapop.net (xx.xxx.xx.xx)  1.945 ms 2.168 ms  2.333 ms
5  et-10-0-0-1275.rtr.atla.net.internet2.edu (xx.xxx.xx.xx) 15.733 ms  33.742 ms  15.99
1 ms
6  et-10-2-0.105.rtr.hous.net.internet2.edu (xx.xxx.xx.xx) 39.407 ms  39.343 ms  39.517 m
s
7  LSUF-1323-L3BR.1045.loni.org (xx.xxx.xx.xx)  45.149 ms 66.910 ms  45.194 ms
8  * * *

Sean