viewtopic.php?f=7&t=2618 points to this:
viewtopic.php?f=7&t=2612
says to put all the info into an email to connection problems.
However, the issue is not exactly a connection issue from this machine.
As i pointed out, I can run the command from command line:
/bin/python $OCSSWROOT/scripts/update_luts.py $1 --verbose
[bmurch@optics0] ~ $
/bin/python $OCSSWROOT/scripts/update_luts.py aqua --verbose
Downloading files into /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/common
...no new files.
Downloading files into /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/modis
+ utcpole.dat
Downloading files into /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/modisa
...no new files.
Downloading files into /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/modisa/cal/OPER
+ MYD02_Emissive_LUTs.V6.2.1.60_OC_v1.60.hdf
+ MYD02_QA_LUTs.V6.2.1.60_OC_v1.60.hdf
+ MYD02_Reflective_LUTs.V6.2.1.60_OC_v1.60.hdf
...deleting outdated LUTs:
- MYD02_Emissive_LUTs.V6.2.1.59_OC_v1.59.hdf
- MYD02_QA_LUTs.V6.2.1.59_OC_v1.59.hdf
- MYD02_Reflective_LUTs.V6.2.1.59_OC_v1.59.hdf
Downloading files into /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/modisa/xcal/OPER
+ xcal_modisa_axc_oc_v1.60d_1240.hdf
...
+ xcal_modisa_axc_oc_v1.60d_869.hdf
...deleting outdated LUTs:
- xcal_modisa_axc_oc_v1.59d_1240.hdf
...
- xcal_modisa_axc_oc_v1.59d_869.hdf
When I run ifconfig commandline:
[bmurch@optics0] ~ $
/usr/sbin/ifconfig
em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.250.0.15 netmask 255.255.248.0 broadcast 10.250.7.255
inet6 fe80::d294:66ff:fe1a:8ad3 prefixlen 64 scopeid 0x20<link>
ether d0:94:66:1a:8a:d3 txqueuelen 1000 (Ethernet)
RX packets 292767146 bytes 46282922721 (43.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 153839597 bytes 34766097790 (32.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 18
em2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether d0:94:66:1a:8a:d4 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 19
em3: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether d0:94:66:1a:8a:d5 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 19
em4: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether d0:94:66:1a:8a:d6 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16
ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
inet 10.250.16.15 netmask 255.255.248.0 broadcast 10.250.23.255
inet6 fe80::211:7500:6e:dd64 prefixlen 64 scopeid 0x20<link>
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
infiniband 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 1024 (InfiniBand)
RX packets 39988663840 bytes 43207908864059 (39.2 TiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 56405773497 bytes 106067394615151 (96.4 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 1378158473 bytes 277831824761 (258.7 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1378158473 bytes 277831824761 (258.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p1p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 131.247.250.109 netmask 255.255.255.192 broadcast 131.247.250.127
inet6 2607:fe50:0:6330::102 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::101 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::100 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330:b696:91ff:fe11:c374 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::112 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::111 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::110 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::109 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::108 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::107 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::106 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::105 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::104 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::103 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::119 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::118 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::117 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::116 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::115 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::114 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::113 prefixlen 64 scopeid 0x0<global>
inet6 fe80::b696:91ff:fe11:c374 prefixlen 64 scopeid 0x20<link>
ether b4:96:91:11:c3:74 txqueuelen 1000 (Ethernet)
RX packets 72795802269 bytes 106084814101698 (96.4 TiB)
RX errors 0 dropped 9450564 overruns 0 frame 0
TX packets 25900869714 bytes 32582684624017 (29.6 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p1p2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether b4:96:91:11:c3:76 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
When I run route:
/usr/sbin/route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 131.247.250.126 0.0.0.0 UG 101 0 0 p1p1
10.250.0.0 0.0.0.0 255.255.248.0 U 100 0 0 em1
10.250.0.0 10.250.7.254 255.255.248.0 UG 100 0 0 em1
10.250.0.204 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.230 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.231 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.232 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.233 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.234 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.235 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.16.0 0.0.0.0 255.255.248.0 U 150 0 0 ib0
131.247.250.64 0.0.0.0 255.255.255.192 U 101 0 0 p1p1
But when I run tracepath I fizzle out on the commandline:
$ /usr/sbin/tracepath oceandata.sci.gsfc.nasa.gov
1?: [LOCALHOST] pmtu 1500
1: vlan250.svc-7700-2.net.usf.edu 0.702ms
1: vlan250.svc-7700-2.net.usf.edu 0.645ms
2: vlan254.campus-backbone1.net.usf.edu 0.882ms
3: vlan256.wan1.net.usf.edu 0.750ms
4: renet-flrcore-108-59-25-196.rtr.net.flrnet.org 2.885ms
5: tlh-flrcore-asr9010-1-hu0600-1.net.flrnet.org 10.024ms asymm 7
6: tlh-flrcore-asr9010-1-hu0600-1.net.flrnet.org 10.069ms asymm 7
7: jax-flrcore-asr9010-1-hu0600-1.net.flrnet.org 10.187ms asymm 6
8: prov-i2-atla-renet-1070.net.flrnet.org 11.458ms asymm 7
9: 163.253.2.32 30.396ms asymm 14
10: 163.253.1.155 30.152ms asymm 13
11: 163.253.2.84 31.380ms
12: et-0-1-8-1275.ashb-core.maxgigapop.net 29.008ms
13: et-11-3-0_5.clpk-core.maxgigapop.net 30.004ms
14: et-11-3-0_5.clpk-core.maxgigapop.net 30.042ms asymm 13
15: vlbi-test1.maxgigapop.net 31.504ms asymm 14
16: no reply
...
30: no reply
Too many hops: pmtu 1500
Resume: pmtu 1500
But, I just ran it again via cron on the same machine (including the ifconfig, /usr/sbin/route -n and /usr/sbin/tracepath oceandata.sci.gsfc.nasa.gov commands):
Cron <bmurch@optics0> /shares/cms_optics/apps/seadas/cron/ocssw_7.5.3-RHEL6.sh aqua
(Cron Daemon) <bmurch@rc.usf.edu>
Tue 10/5/2021 12:04 PM
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw
/bin/python /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/scripts/update_luts.py aqua --verbose
Downloading files into /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/common
Exception: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
Exception: 'NoneType' object has no attribute 'status'
Traceback (most recent call last):
File "/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/scripts/update_luts.py", line 64, in <module>
luts.get_luts()
File "/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/scripts/modules/LutUtils.py", line 134, in get_luts
check_times=True)
File "/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/scripts/modules/JsonUtils.py", line 272, in download_allfiles
all_links = self.get_links(url, regex=regex)
File "/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/scripts/modules/JsonUtils.py", line 238, in get_links
content = response.read()
AttributeError: 'NoneType' object has no attribute 'read'
/usr/sbin/ifconfig
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.250.0.15 netmask 255.255.248.0 broadcast 10.250.7.255
inet6 fe80::d294:66ff:fe1a:8ad3 prefixlen 64 scopeid 0x20<link>
ether d0:94:66:1a:8a:d3 txqueuelen 1000 (Ethernet)
RX packets 292785544 bytes 46285502966 (43.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 153847514 bytes 34768314121 (32.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 18
em2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether d0:94:66:1a:8a:d4 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 19
em3: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether d0:94:66:1a:8a:d5 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 19
em4: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether d0:94:66:1a:8a:d6 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16
ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
inet 10.250.16.15 netmask 255.255.248.0 broadcast 10.250.23.255
inet6 fe80::211:7500:6e:dd64 prefixlen 64 scopeid 0x20<link>
infiniband 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 1024 (InfiniBand)
RX packets 39988816553 bytes 43207923083096 (39.2 TiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 56406056396 bytes 106067830876104 (96.4 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 1378187309 bytes 277836660132 (258.7 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1378187309 bytes 277836660132 (258.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p1p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 131.247.250.109 netmask 255.255.255.192 broadcast 131.247.250.127
inet6 2607:fe50:0:6330::102 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::101 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::100 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330:b696:91ff:fe11:c374 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::112 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::111 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::110 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::109 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::108 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::107 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::106 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::105 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::104 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::103 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::119 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::118 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::117 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::116 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::115 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::114 prefixlen 64 scopeid 0x0<global>
inet6 2607:fe50:0:6330::113 prefixlen 64 scopeid 0x0<global>
inet6 fe80::b696:91ff:fe11:c374 prefixlen 64 scopeid 0x20<link>
ether b4:96:91:11:c3:74 txqueuelen 1000 (Ethernet)
RX packets 72796549624 bytes 106085880249405 (96.4 TiB)
RX errors 0 dropped 9451158 overruns 0 frame 0
TX packets 25901626362 bytes 32583740026243 (29.6 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p1p2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether b4:96:91:11:c3:76 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
/usr/sbin/route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 131.247.250.126 0.0.0.0 UG 101 0 0 p1p1
10.250.0.0 0.0.0.0 255.255.248.0 U 100 0 0 em1
10.250.0.0 10.250.7.254 255.255.248.0 UG 100 0 0 em1
10.250.0.204 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.230 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.231 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.232 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.233 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.234 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.0.235 131.247.250.126 255.255.255.255 UGH 0 0 0 p1p1
10.250.16.0 0.0.0.0 255.255.248.0 U 150 0 0 ib0
131.247.250.64 0.0.0.0 255.255.255.192 U 101 0 0 p1p1
/usr/sbin/tracepath oceandata.sci.gsfc.nasa.gov
1?: [LOCALHOST] pmtu 1500
1: vlan250.svc-7700-2.net.usf.edu 0.704ms
1: vlan250.svc-7700-2.net.usf.edu 0.674ms
2: vlan254.campus-backbone2.net.usf.edu 0.908ms
3: Vlan257.wan2.net.usf.edu 0.745ms
4: renet-flrcore-108-59-25-196.rtr.net.flrnet.org 2.803ms
5: renet-flrcore-108-59-25-196.rtr.net.flrnet.org 2.673ms asymm 4
6: jax-flrcore-asr9010-1-hu0600-1.net.flrnet.org 10.142ms
7: prov-i2-atla-renet-1070.net.flrnet.org 13.025ms
8: 163.253.2.32 31.168ms asymm 14
9: 163.253.2.32 31.859ms asymm 14
10: no reply
11: 163.253.2.84 30.813ms
12: et-0-1-8-1275.ashb-core.maxgigapop.net 28.896ms
13: et-0-1-8-1275.ashb-core.maxgigapop.net 34.417ms asymm 12
14: et-11-3-0_5.clpk-core.maxgigapop.net 30.059ms asymm 13
15: vlbi-test1.maxgigapop.net 32.436ms asymm 14
16: no reply
17: no reply
18: no reply
19: no reply
20: no reply
21: no reply
22: no reply
23: no reply
24: no reply
25: no reply
26: no reply
27: no reply
28: no reply
29: no reply
30: no reply
Too many hops: pmtu 1500
Resume: pmtu 1500
Furthermore, it also says
viewtopic.php?f=7&t=2612#p9314
"First make sure that you have applied all available software updates and security patches to your own systems. A reboot may be required."
But it works with on the commandline, but not via cron.
As this is a shared SLURM node in the computing center, I cannot make changes to the OS in anyway.
Note, this is a CentOS7 machine. This exact script still runs fine on another older CentOS6 machine.
Please advise.
Thanks,
Brock