Earthdata Forum

After a period of relative stability using SeaDAS 7.4, my connectivity issues have returned (sometime late Monday morning, central time).

I a trying to install SeaDAS on each of a half a dozen graduate student accounts on Linux workstations (CentOS 7). My own installation had been working fine.

The installer shell script works fine. It's when I use the GUI to install the OCSSW processors that behavior gets weird.

Each attempt to install a processor results in a popup that says "resolving oceandata.sci..." that lingers for several minutes. A "host" command resolves the site just fine. Eventually downloads begin and in some cases the processor succeeds, and others fail.The failure report shows a connection timeout, and then success on a second try for "bundles.sha256sum", but then fails on the common bundle.

Running the manual installer also spends a long time trying to connect to the IPv6 address.

Curiously/notably, I can traceroute to oceandata.sci... but traceroute6 does not work, immediately listing a series of "* * *" lines.

The campus ITS staff are seeing this too, from an entirely different building, and even from the border net.

As of this morning their results are:


H:\>tracert oceandata.sci.gsfc.nasa.gov

Tracing route to gs6142obpg1020.sci.gsfc.nasa.gov [2001:4d0:2418:128::84]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  2620:105:b007:3103::1
  2     5 ms    <1 ms    <1 ms  2620:105:b000::1e
  3    <1 ms    <1 ms    <1 ms  2620:105:b000:0:2:4:8:2
  4     1 ms     1 ms     1 ms  2620:105:b000::fa
  5    10 ms     1 ms     1 ms  2607:f390:0:1001::89
  6     1 ms    <1 ms     1 ms  LSUF-R2-ET-7-3-0.loni.org [2607:f390:0:1000::2]

  7     7 ms     7 ms     7 ms  LSUF-R2-ET-7-1-0.loni.org [2607:f390:0:1001::6e]

  8    30 ms    30 ms    31 ms  et-3-3-0.4070.rtsw.atla.net.internet2.edu [2001:
468:ff:100::1]
  9    44 ms    44 ms    44 ms  atlang-obslan.abilene.ucaid.edu [2001:468:e:3::1
]
10    46 ms    45 ms    48 ms  2001:468:c00:11::1
11    48 ms    48 ms    48 ms  2001:468:c00:fff0::2
12     *        *        *     Request timed out.
13     *        *        *     Request timed out.
14     *        *        *     Request timed out.
15     *        *        *     Request timed out.
16     *        *        *     Request timed out.


Are you having any issues with ISP? Any other light you can shed is greatly appreciated!

Alaric

Aleric,

From your post it seems like connectivity works, but stalls or fails during the download. Can you verify DNS and basic connectivity for me? The addition of the "time" command will give an idea of how fast the system and network services are responding.

On the Linux workstations, does DNS resolution work?

$ time host oceandata.sci.gsfc.nasa.gov
oceandata.sci.gsfc.nasa.gov is an alias for gs6142obpg1020.sci.gsfc.nasa.gov.
gs6142obpg1020.sci.gsfc.nasa.gov has address xx.xxx.xx.xx
gs6142obpg1020.sci.gsfc.nasa.gov has IPv6 address 2001:4d0:2418:128::84
gs6142obpg1020.sci.gsfc.nasa.gov mail is handled by 10 oceanpost.sci.gsfc.nasa.gov.
real    0m0.002s

Can you connect to oceandata using ipv4 and ipv6?

$  time curl -4 -kIL https://oceandata.sci.gsfc.nasa.gov
HTTP/1.1 200 OK
real    0m0.046s

$  time curl -6 -kIL https://oceandata.sci.gsfc.nasa.gov
HTTP/1.1 200 OK
real    0m0.067s

Many thanks!

I thought the traceroute from here (by my campus IT folks) seemed to stall at gigapop, without reaching you, didn't it?

The results below show IPv6 has an issue...

Anyhow, I have tried disabling IPv6 (and my firewall just to be sure!) and rebooting one of the worksatations here and still can't get a response from a manual install (much less a GUI install). What's also curious is that after several retries among the 8 workstations in this room over the past day or two, some were eventually able to install a suite of about 6 processors. But some will not.They did "linger" with a "resolving oceandata..." popup, but the "host" command has been responsive throughout those "lingers". But connecting via IPv6 stalls.


$ time host oceandata.sci.gsfc.nasa.gov
oceandata.sci.gsfc.nasa.gov is an alias for gs6142obpg1020.sci.gsfc.nasa.gov.
gs6142obpg1020.sci.gsfc.nasa.gov has address xx.xxx.xx.xx
gs6142obpg1020.sci.gsfc.nasa.gov has IPv6 address 2001:4d0:2418:128::84
gs6142obpg1020.sci.gsfc.nasa.gov mail is handled by 10 oceanpost.sci.gsfc.nasa.gov.

real  0m3.239s
user  0m0.012s
sys  0m0.012s

$ time curl -4 -kIL https://oceandata.sci.gsfc.nasa.gov
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 23 Mar 2017 17:37:49 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 12694
Connection: keep-alive
Set-Cookie: app-obdaac=725592d12d717e3461abaa1fb20dab14095fc862--ed605283767f24a83bdd1aed512ab432ce426d0e; expires=Thu, 23 Mar 2017 18:37:49 GMT; path=/; secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

real  0m6.155s
user  0m0.106s
sys  0m0.074s

$ time curl -6 -kIL https://oceandata.sci.gsfc.nasa.gov
curl: (7) Failed connect to oceandata.sci.gsfc.nasa.gov:443; Connection timed out

real  2m12.815s
user  0m0.003s
sys  0m0.012s


Looks like ipv6 connectivity is the problem. Blocking ipv6 on the firewall will still allow the application to try to connect using ipv6, then have to timeout on each attempt. If you removed the ipv6 address from the workstation the system would only use ipv4 and the install should work without issue.

We verified our our side that your ipv4 and ipv6 addresses are not blocked. I am going to contact another network group at GSFC to see if they have any ideas about your ipv6 traceroute.

Disabling IPv6 ends up not being an option, as it breaks other services within my lab. The folks at my NOC think they are seeing some routing issues, and are going to try to reach out to the same group at your end to discuss what they're seeing.

Many thanks!

Hello.

I have a similar issue with my station. Here are my results

$ time curl -4 -kIL https://oceandata.sci.gsfc.nasa.gov
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 18 May 2017 06:18:54 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 12663
Connection: keep-alive
Set-Cookie: app-obdaac=0cd75d7f0bbd9f2e33b88751feba3df512335a03--138133edd36665f30675cfd1be2182e585b5ee9d; expires=Thu, 18 May 2017 07:18:54 GMT; path=/; secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

real  0m1.727s
user  0m0.064s
sys  0m0.048s

$ time curl -6 -kIL https://oceandata.sci.gsfc.nasa.gov
curl: (7) Couldn't connect to server

real  0m0.261s
user  0m0.008s
sys  0m0.000s

$ time host oceandata.sci.gsfc.nasa.gov
oceandata.sci.gsfc.nasa.gov is an alias for gs6142obpg1020.sci.gsfc.nasa.gov.
gs6142obpg1020.sci.gsfc.nasa.gov has address xx.xxx.xx.xx
gs6142obpg1020.sci.gsfc.nasa.gov has IPv6 address 2001:4d0:2418:128::84
gs6142obpg1020.sci.gsfc.nasa.gov mail is handled by 10 oceanpost.sci.gsfc.nasa.gov.

real  0m0.265s
user  0m0.008s
sys  0m0.004s

Is it probably the reason why I cannot execute L1A to L2 processing and is it possible to solve my problem?

Regards,
Kim

Kim,

If the computer is preferring IPv6 and not falling back to IPv4 then you might be experiencing the ipv6 issue.

First, can you get to google using IPv6?

       curl -6 -kIL https://www.google.com

One option is disable the IPv6 ip on the local system this will force the network to use IPv4.

You may also want to ask you local IT support staff to check connectivity. They can go thorough the steps on the following link:

    Having Data Access Issues?
    https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6447