两台IB交换机使用server side方式做高可用的记录

两台IB交换机使用server side方式做高可用的记录

额外注意：IB交换机也分可不可以管理，比如SX6036就是可以管理的IB交换机，
两台可以管理的IB交换机可以在IB交换机上配置高可用来实现其中一台IB交换机意外离线而导致的IB网络失效的问题。

若是不可管理的IB交换机，就只能在服务器端使用opensm来实现IB网络的高可用性。

介绍：
三台pcserver，安装RHEL7.2，每台pcserver上安装有两块HCA卡，每块HCA卡2个端口（port），
只不过每块HCA卡的第二个端口（port）没有插入IB线缆，并且每块HCA卡的第二个端口（port）在nmtui中被disable掉了。
三台主机的主机名为
rac1
rac2
node1


两台IB交换机，SX6036。

在三台pcserver上，安装mellanox的ofed驱动，ofed驱动使用opensmd服务来管理IB的子网。如下所示：

[root@rac1 ~]# systemctl status opensmd● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager   Loaded: loaded (/etc/rc.d/init.d/opensmd)   Active: active (running) since Mon 2017-06-05 18:39:07 CST; 14h ago     Docs: man:systemd-sysv-generator(8)  Process: 2762 ExecStop=/etc/rc.d/init.d/opensmd stop (code=exited, status=0/SUCCESS)  Process: 2786 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS) Main PID: 2793 (opensm)   CGroup: /system.slice/opensmd.service           └─2793 /usr/sbin/opensm --daemonJun 05 18:39:06 rac1 OpenSM[2791]:  Loading Cached Option:guid = 0x7cfe900300917f11Jun 05 18:39:06 rac1 OpenSM[2791]:  Loading Cached Option:guid = 0x7cfe900300917ed1Jun 05 18:39:06 rac1 OpenSM[2793]: /var/log/opensm.log log file openedJun 05 18:39:06 rac1 OpenSM[2793]: OpenSM 4.8.1.MLNX20170118.1a8ad26Jun 05 18:39:06 rac1 OpenSM[2793]: Entering DISCOVERING stateJun 05 18:39:06 rac1 OpenSM[2793]: Entering STANDBY stateJun 05 18:39:07 rac1 opensmd[2786]: Starting IB Subnet Manager.[  OK  ]Jun 05 18:39:07 rac1 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager.Jun 05 18:43:06 rac1 OpenSM[2793]: Entering DISCOVERING stateJun 05 18:43:06 rac1 OpenSM[2793]: Entering STANDBY state[root@rac1 ~]#

[root@rac2 ~]# systemctl status opensmd● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager   Loaded: loaded (/etc/rc.d/init.d/opensmd)   Active: active (running) since Mon 2017-06-05 18:48:07 CST; 14h ago     Docs: man:systemd-sysv-generator(8)  Process: 1679 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS) Main PID: 1711 (opensm)   CGroup: /system.slice/opensmd.service           └─1711 /usr/sbin/opensm --daemonJun 05 18:48:06 rac2 OpenSM[1695]:  Loading Cached Option:guid = 0x248a0703005d07f1Jun 05 18:48:06 rac2 OpenSM[1695]:  Loading Cached Option:guid = 0x248a0703005d0841Jun 05 18:48:06 rac2 OpenSM[1711]: /var/log/opensm.log log file openedJun 05 18:48:06 rac2 OpenSM[1711]: OpenSM 4.8.1.MLNX20170118.1a8ad26Jun 05 18:48:06 rac2 OpenSM[1711]: Entering DISCOVERING stateJun 05 18:48:06 rac2 OpenSM[1711]: SM port is downJun 05 18:48:07 rac2 opensmd[1679]: Starting IB Subnet Manager.[  OK  ]Jun 05 18:48:07 rac2 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager.Jun 05 18:48:46 rac2 OpenSM[1711]: SM port is upJun 05 18:48:46 rac2 OpenSM[1711]: Entering STANDBY state[root@rac2 ~]# 

[root@host1 ~]# systemctl status opensmd● opensmd.service - LSB: Activates/Deactivates InfiniBand Subnet Manager   Loaded: loaded (/etc/rc.d/init.d/opensmd)   Active: active (running) since Mon 2017-06-05 18:33:50 CST; 14h ago     Docs: man:systemd-sysv-generator(8)  Process: 2935 ExecStart=/etc/rc.d/init.d/opensmd start (code=exited, status=0/SUCCESS) Main PID: 2941 (opensm)   CGroup: /system.slice/opensmd.service           └─2941 /usr/sbin/opensm --daemonJun 05 18:33:49 host1 systemd[1]: Starting LSB: Activates/Deactivates InfiniBand Subnet Manager...Jun 05 18:33:49 host1 OpenSM[2941]: /var/log/opensm.log log file openedJun 05 18:33:49 host1 OpenSM[2941]: OpenSM 4.8.1.MLNX20170118.1a8ad26Jun 05 18:33:49 host1 OpenSM[2941]: Entering DISCOVERING stateJun 05 18:33:49 host1 OpenSM[2941]: Entering STANDBY stateJun 05 18:33:50 host1 opensmd[2935]: Starting IB Subnet Manager.[  OK  ]Jun 05 18:33:50 host1 systemd[1]: Started LSB: Activates/Deactivates InfiniBand Subnet Manager.Jun 05 18:43:09 host1 OpenSM[2941]: Entering DISCOVERING stateJun 05 18:43:09 host1 OpenSM[2941]: Entering MASTER state[root@host1 ~]#

三台主机的/etc/opensm/opensm.conf文件的内容如下（只贴自行修改的部分）：

rac1上的/etc/opensm/opensm.conf文件的内容：# The port GUID on which the OpenSM is runningguid 0x7cfe900300917f11guid 0x7cfe900300917ed1

rac2上的/etc/opensm/opensm.conf文件的内容：# The port GUID on which the OpenSM is runningguid 0x248a0703005d07f1guid 0x248a0703005d0841

rac2上的/etc/opensm/opensm.conf文件的内容：# The port GUID on which the OpenSM is runningguid 0x248a0703005d07f1guid 0x248a0703005d0841

rac1的ip地址如下：

[root@rac1 ~]# ip addr show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether 0c:c4:7a:ac:3c:c8 brd ff:ff:ff:ff:ff:ff    inet 192.168.100.101/24 brd 192.168.100.255 scope global enp5s0f0       valid_lft forever preferred_lft forever    inet6 fe80::ec4:7aff:feac:3cc8/64 scope link       valid_lft forever preferred_lft forever3: enp5s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether 0c:c4:7a:ac:3c:c9 brd ff:ff:ff:ff:ff:ff    inet 172.17.0.1/24 brd 172.17.0.255 scope global enp5s0f1       valid_lft forever preferred_lft forever    inet6 fe80::ec4:7aff:feac:3cc9/64 scope link       valid_lft forever preferred_lft forever4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:11 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff    inet 10.10.10.101/24 brd 10.10.10.255 scope global ib0       valid_lft forever preferred_lft forever    inet6 fe80::7efe:9003:91:7f11/64 scope link       valid_lft forever preferred_lft forever5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:12 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff    inet 10.10.11.102/24 brd 10.10.11.255 scope global ib2       valid_lft forever preferred_lft forever    inet6 fe80::7efe:9003:91:7ed1/64 scope link       valid_lft forever preferred_lft forever7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN    link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0       valid_lft forever preferred_lft forever9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500    link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff[root@rac1 ~]#

rac2的ip地址如下：

[root@rac2 ~]# ip addr show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether 0c:c4:7a:ab:d3:4c brd ff:ff:ff:ff:ff:ff    inet 192.168.100.102/24 brd 192.168.100.255 scope global enp5s0f0       valid_lft forever preferred_lft forever    inet6 fe80::ec4:7aff:feab:d34c/64 scope link       valid_lft forever preferred_lft forever3: enp5s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000    link/ether 0c:c4:7a:ab:d3:4d brd ff:ff:ff:ff:ff:ff4: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:07:f1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff    inet 10.10.10.105/24 brd 10.10.10.255 scope global ib0       valid_lft forever preferred_lft forever5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:07:f2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:08:41 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff    inet 10.10.11.107/24 brd 10.10.11.255 scope global ib2       valid_lft forever preferred_lft forever    inet6 fe80::268a:703:5d:841/64 scope link       valid_lft forever preferred_lft forever7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:24:8a:07:03:00:5d:08:42 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN    link/ether 52:54:00:74:ff:7b brd ff:ff:ff:ff:ff:ff    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0       valid_lft forever preferred_lft forever9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500    link/ether 52:54:00:74:ff:7b brd ff:ff:ff:ff:ff:ff[root@rac2 ~]# 

node1（第三台pcsever）的ip地址如下：

[root@host1 ~]# ip addr show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether 0c:c4:7a:ab:e5:ce brd ff:ff:ff:ff:ff:ff    inet 192.168.100.103/24 brd 192.168.100.255 scope global enp5s0f0       valid_lft forever preferred_lft forever    inet6 fe80::ec4:7aff:feab:e5ce/64 scope link       valid_lft forever preferred_lft forever3: enp5s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000    link/ether 0c:c4:7a:ab:e5:cf brd ff:ff:ff:ff:ff:ff4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:16:c2:31 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff    inet 10.10.10.109/24 brd 10.10.10.255 scope global ib0       valid_lft forever preferred_lft forever    inet6 fe80::7efe:9003:16:c231/64 scope link       valid_lft forever preferred_lft forever5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:16:c2:32 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7d:a1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff    inet 10.10.11.111/24 brd 10.10.11.255 scope global ib2       valid_lft forever preferred_lft forever    inet6 fe80::7efe:9003:91:7da1/64 scope link       valid_lft forever preferred_lft forever7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7d:a2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN    link/ether 52:54:00:53:65:c8 brd ff:ff:ff:ff:ff:ff    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0       valid_lft forever preferred_lft forever9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500    link/ether 52:54:00:53:65:c8 brd ff:ff:ff:ff:ff:ff[root@host1 ~]# 

rac1:

[root@rac1 ~]# ibhostsCa    : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"Ca    : 0x7cfe90030016c230 ports 2 "host1 HCA-1"Ca    : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"[root@rac1 ~]# ibaddrGID fe80::7cfe:9003:91:7f11 LID start 0x2 end 0x2[root@rac1 ~]# ibstatCA 'mlx4_0'    CA type: MT4099    Number of ports: 2    Firmware version: 2.40.7000    Hardware version: 1    Node GUID: 0x7cfe900300917f10    System image GUID: 0x7cfe900300917f13    Port 1:        State: Active        Physical state: LinkUp        Rate: 56        Base lid: 2        LMC: 0        SM lid: 4        Capability mask: 0x02514868        Port GUID: 0x7cfe900300917f11        Link layer: InfiniBand    Port 2:        State: Down        Physical state: Polling        Rate: 10        Base lid: 0        LMC: 0        SM lid: 0        Capability mask: 0x02514868        Port GUID: 0x7cfe900300917f12        Link layer: InfiniBandCA 'mlx4_1'    CA type: MT4099    Number of ports: 2    Firmware version: 2.40.7000    Hardware version: 1    Node GUID: 0x7cfe900300917ed0    System image GUID: 0x7cfe900300917ed3    Port 1:        State: Active        Physical state: LinkUp        Rate: 56        Base lid: 5        LMC: 0        SM lid: 8        Capability mask: 0x0251486a        Port GUID: 0x7cfe900300917ed1        Link layer: InfiniBand    Port 2:        State: Down        Physical state: Polling        Rate: 10        Base lid: 0        LMC: 0        SM lid: 0        Capability mask: 0x02514868        Port GUID: 0x7cfe900300917ed2        Link layer: InfiniBand[root@rac1 ~]# ibstatusInfiniband device 'mlx4_0' port 1 status:    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7f11    base lid:     0x2    sm lid:         0x4    state:         4: ACTIVE    phys state:     5: LinkUp    rate:         56 Gb/sec (4X FDR)    link_layer:     InfiniBandInfiniband device 'mlx4_0' port 2 status:    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7f12    base lid:     0x0    sm lid:         0x0    state:         1: DOWN    phys state:     2: Polling    rate:         10 Gb/sec (4X)    link_layer:     InfiniBandInfiniband device 'mlx4_1' port 1 status:    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7ed1    base lid:     0x5    sm lid:         0x8    state:         4: ACTIVE    phys state:     5: LinkUp    rate:         56 Gb/sec (4X FDR)    link_layer:     InfiniBandInfiniband device 'mlx4_1' port 2 status:    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7ed2    base lid:     0x0    sm lid:         0x0    state:         1: DOWN    phys state:     2: Polling    rate:         10 Gb/sec (4X)    link_layer:     InfiniBand[root@rac1 ~]# ibv_ibv_asyncwatch     ibv_dcini          ibv_devices        ibv_intf           ibv_rc_pingpong    ibv_srq_pingpong   ibv_uc_pingpong    ibv_umribv_cc_pingpong    ibv_dctgt          ibv_devinfo        ibv_polldcinfo     ibv_shared_mr      ibv_task_pingpong  ibv_ud_pingpong    ibv_xsrq_pingpong[root@rac1 ~]# ibv_devinfohca_id:    mlx4_1    transport:            InfiniBand (0)    fw_ver:                2.40.7000    node_guid:            7cfe:9003:0091:7ed0    sys_image_guid:            7cfe:9003:0091:7ed3    vendor_id:            0x02c9    vendor_part_id:            4099    hw_ver:                0x1    board_id:            MT_1090120019    phys_port_cnt:            2    Device ports:        port:    1            state:            PORT_ACTIVE (4)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            8            port_lid:        5            port_lmc:        0x00            link_layer:        InfiniBand        port:    2            state:            PORT_DOWN (1)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            0            port_lid:        0            port_lmc:        0x00            link_layer:        InfiniBandhca_id:    mlx4_0    transport:            InfiniBand (0)    fw_ver:                2.40.7000    node_guid:            7cfe:9003:0091:7f10    sys_image_guid:            7cfe:9003:0091:7f13    vendor_id:            0x02c9    vendor_part_id:            4099    hw_ver:                0x1    board_id:            MT_1090120019    phys_port_cnt:            2    Device ports:        port:    1            state:            PORT_ACTIVE (4)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            4            port_lid:        2            port_lmc:        0x00            link_layer:        InfiniBand        port:    2            state:            PORT_DOWN (1)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            0            port_lid:        0            port_lmc:        0x00            link_layer:        InfiniBand[root@rac1 ~]# ibnetdiscover## Topology file: generated on Tue Jun  6 08:37:33 2017## Initiated from node 7cfe900300917f10 port 7cfe900300917f11vendid=0x2c9devid=0xc738sysimgguid=0x7cfe900300f17940switchguid=0x7cfe900300f17940(7cfe900300f17940)Switch    36 "S-7cfe900300f17940"        # "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0[31]    "H-7cfe90030016c230"[1](7cfe90030016c231)         # "host1 HCA-1" lid 3 4xFDR[33]    "H-248a0703005d07f0"[1](248a0703005d07f1)         # "rac2 HCA-1" lid 0 4xFDR[35]    "H-7cfe900300917f10"[1](7cfe900300917f11)         # "rac1 HCA-1" lid 2 4xFDRvendid=0x2c9devid=0x1003sysimgguid=0x248a0703005d07f3caguid=0x248a0703005d07f0Ca    2 "H-248a0703005d07f0"        # "rac2 HCA-1"[1](248a0703005d07f1)     "S-7cfe900300f17940"[33]        # lid 0 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDRvendid=0x2c9devid=0x1003sysimgguid=0x7cfe90030016c233caguid=0x7cfe90030016c230Ca    2 "H-7cfe90030016c230"        # "host1 HCA-1"[1](7cfe90030016c231)     "S-7cfe900300f17940"[31]        # lid 3 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDRvendid=0x2c9devid=0x1003sysimgguid=0x7cfe900300917f13caguid=0x7cfe900300917f10Ca    2 "H-7cfe900300917f10"        # "rac1 HCA-1"[1](7cfe900300917f11)     "S-7cfe900300f17940"[35]        # lid 2 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR[root@rac1 ~]# iblinkinfoCA: rac2 HCA-1:      0x248a0703005d07f1      0    1[  ] ==( 4X       14.0625 Gbps Initialize/  LinkUp)==>       1   33[  ] "MF0;switch-138276:SX6036/U1" ( )CA: host1 HCA-1:      0x7cfe90030016c231      3    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1   31[  ] "MF0;switch-138276:SX6036/U1" ( )Switch: 0x7cfe900300f17940 MF0;switch-138276:SX6036/U1:           1    1[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    2[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    3[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    9[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   10[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   11[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   12[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   13[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   14[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   15[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   16[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   17[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   18[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   19[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   20[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   21[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   22[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   23[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   24[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   25[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   26[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   27[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   28[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   29[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   30[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   31[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       3    1[  ] "host1 HCA-1" ( )           1   32[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   33[  ] ==( 4X       14.0625 Gbps Initialize/  LinkUp)==>       0    1[  ] "rac2 HCA-1" ( )           1   34[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   35[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       2    1[  ] "rac1 HCA-1" ( )           1   36[  ] ==(                Down/ Polling)==>             [  ] "" ( )CA: rac1 HCA-1:      0x7cfe900300917f11      2    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1   35[  ] "MF0;switch-138276:SX6036/U1" ( )[root@rac1 ~]# ibswitchesSwitch    : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0[root@rac1 ~]# ibnodesCa    : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"Ca    : 0x7cfe90030016c230 ports 2 "host1 HCA-1"Ca    : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"Switch    : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0[root@rac1 ~]# 

rac2:

[root@rac2 ~]# ibhostsCa    : 0x7cfe900300917ed0 ports 2 "rac1 HCA-2"Ca    : 0x7cfe900300917da0 ports 2 "host1 HCA-2"Ca    : 0x248a0703005d0840 ports 2 "rac2 HCA-2"[root@rac2 ~]# ibaddrGID fe80::248a:703:5d:841 LID start 0x1 end 0x1[root@rac2 ~]# ibstatCA 'mlx4_0'    CA type: MT4099    Number of ports: 2    Firmware version: 2.40.7000    Hardware version: 1    Node GUID: 0x248a0703005d07f0    System image GUID: 0x248a0703005d07f3    Port 1:        State: Initializing        Physical state: LinkUp        Rate: 56        Base lid: 0        LMC: 0        SM lid: 0        Capability mask: 0x02514868        Port GUID: 0x248a0703005d07f1        Link layer: InfiniBand    Port 2:        State: Down        Physical state: Polling        Rate: 10        Base lid: 0        LMC: 0        SM lid: 0        Capability mask: 0x02514868        Port GUID: 0x248a0703005d07f2        Link layer: InfiniBandCA 'mlx4_1'    CA type: MT4099    Number of ports: 2    Firmware version: 2.40.7000    Hardware version: 1    Node GUID: 0x248a0703005d0840    System image GUID: 0x248a0703005d0843    Port 1:        State: Active        Physical state: LinkUp        Rate: 56        Base lid: 1        LMC: 0        SM lid: 8        Capability mask: 0x0251486a        Port GUID: 0x248a0703005d0841        Link layer: InfiniBand    Port 2:        State: Down        Physical state: Polling        Rate: 10        Base lid: 0        LMC: 0        SM lid: 0        Capability mask: 0x02514868        Port GUID: 0x248a0703005d0842        Link layer: InfiniBand[root@rac2 ~]# ibstatusInfiniband device 'mlx4_0' port 1 status:    default gid:     fe80:0000:0000:0000:248a:0703:005d:07f1    base lid:     0x0    sm lid:         0x0    state:         2: INIT    phys state:     5: LinkUp    rate:         56 Gb/sec (4X FDR)    link_layer:     InfiniBandInfiniband device 'mlx4_0' port 2 status:    default gid:     fe80:0000:0000:0000:248a:0703:005d:07f2    base lid:     0x0    sm lid:         0x0    state:         1: DOWN    phys state:     2: Polling    rate:         10 Gb/sec (4X)    link_layer:     InfiniBandInfiniband device 'mlx4_1' port 1 status:    default gid:     fe80:0000:0000:0000:248a:0703:005d:0841    base lid:     0x1    sm lid:         0x8    state:         4: ACTIVE    phys state:     5: LinkUp    rate:         56 Gb/sec (4X FDR)    link_layer:     InfiniBandInfiniband device 'mlx4_1' port 2 status:    default gid:     fe80:0000:0000:0000:248a:0703:005d:0842    base lid:     0x0    sm lid:         0x0    state:         1: DOWN    phys state:     2: Polling    rate:         10 Gb/sec (4X)    link_layer:     InfiniBand[root@rac2 ~]# ibv_devinfohca_id:    mlx4_1    transport:            InfiniBand (0)    fw_ver:                2.40.7000    node_guid:            248a:0703:005d:0840    sys_image_guid:            248a:0703:005d:0843    vendor_id:            0x02c9    vendor_part_id:            4099    hw_ver:                0x1    board_id:            MT_1090120019    phys_port_cnt:            2    Device ports:        port:    1            state:            PORT_ACTIVE (4)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            8            port_lid:        1            port_lmc:        0x00            link_layer:        InfiniBand        port:    2            state:            PORT_DOWN (1)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            0            port_lid:        0            port_lmc:        0x00            link_layer:        InfiniBandhca_id:    mlx4_0    transport:            InfiniBand (0)    fw_ver:                2.40.7000    node_guid:            248a:0703:005d:07f0    sys_image_guid:            248a:0703:005d:07f3    vendor_id:            0x02c9    vendor_part_id:            4099    hw_ver:                0x1    board_id:            MT_1090120019    phys_port_cnt:            2    Device ports:        port:    1            state:            PORT_INIT (2)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            0            port_lid:        0            port_lmc:        0x00            link_layer:        InfiniBand        port:    2            state:            PORT_DOWN (1)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            0            port_lid:        0            port_lmc:        0x00            link_layer:        InfiniBand[root@rac2 ~]# ibnetdiscover## Topology file: generated on Tue Jun  6 08:36:34 2017## Initiated from node 248a0703005d0840 port 248a0703005d0841vendid=0x2c9devid=0xc738sysimgguid=0x7cfe900300ef40c0switchguid=0x7cfe900300ef40c0(7cfe900300ef40c0)Switch    36 "S-7cfe900300ef40c0"        # "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0[31]    "H-7cfe900300917da0"[1](7cfe900300917da1)         # "host1 HCA-2" lid 8 4xFDR[33]    "H-248a0703005d0840"[1](248a0703005d0841)         # "rac2 HCA-2" lid 1 4xFDR[35]    "H-7cfe900300917ed0"[1](7cfe900300917ed1)         # "rac1 HCA-2" lid 5 4xFDRvendid=0x2c9devid=0x1003sysimgguid=0x7cfe900300917ed3caguid=0x7cfe900300917ed0Ca    2 "H-7cfe900300917ed0"        # "rac1 HCA-2"[1](7cfe900300917ed1)     "S-7cfe900300ef40c0"[35]        # lid 5 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDRvendid=0x2c9devid=0x1003sysimgguid=0x7cfe900300917da3caguid=0x7cfe900300917da0Ca    2 "H-7cfe900300917da0"        # "host1 HCA-2"[1](7cfe900300917da1)     "S-7cfe900300ef40c0"[31]        # lid 8 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDRvendid=0x2c9devid=0x1003sysimgguid=0x248a0703005d0843caguid=0x248a0703005d0840Ca    2 "H-248a0703005d0840"        # "rac2 HCA-2"[1](248a0703005d0841)     "S-7cfe900300ef40c0"[33]        # lid 1 lmc 0 "MF0;switch-cad412:SX6036/U1" lid 7 4xFDR[root@rac2 ~]# iblinkinfoCA: rac1 HCA-2:      0x7cfe900300917ed1      5    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       7   35[  ] "MF0;switch-cad412:SX6036/U1" ( )CA: host1 HCA-2:      0x7cfe900300917da1      8    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       7   31[  ] "MF0;switch-cad412:SX6036/U1" ( )Switch: 0x7cfe900300ef40c0 MF0;switch-cad412:SX6036/U1:           7    1[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7    2[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7    3[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7    9[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   10[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   11[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   12[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   13[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   14[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   15[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   16[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   17[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   18[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   19[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   20[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   21[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   22[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   23[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   24[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   25[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   26[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   27[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   28[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   29[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   30[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   31[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       8    1[  ] "host1 HCA-2" ( )           7   32[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   33[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1    1[  ] "rac2 HCA-2" ( )           7   34[  ] ==(                Down/ Polling)==>             [  ] "" ( )           7   35[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       5    1[  ] "rac1 HCA-2" ( )           7   36[  ] ==(                Down/ Polling)==>             [  ] "" ( )CA: rac2 HCA-2:      0x248a0703005d0841      1    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       7   33[  ] "MF0;switch-cad412:SX6036/U1" ( )[root@rac2 ~]# ibswitchesSwitch    : 0x7cfe900300ef40c0 ports 36 "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0[root@rac2 ~]# ibnodesCa    : 0x7cfe900300917ed0 ports 2 "rac1 HCA-2"Ca    : 0x7cfe900300917da0 ports 2 "host1 HCA-2"Ca    : 0x248a0703005d0840 ports 2 "rac2 HCA-2"Switch    : 0x7cfe900300ef40c0 ports 36 "MF0;switch-cad412:SX6036/U1" enhanced port 0 lid 7 lmc 0[root@rac2 ~]# 

node1:

[root@host1 ~]# ibhostsCa    : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"Ca    : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"Ca    : 0x7cfe90030016c230 ports 2 "host1 HCA-1"[root@host1 ~]# ibaddrGID fe80::7cfe:9003:16:c231 LID start 0x3 end 0x3[root@host1 ~]# ibstatCA 'mlx4_0'    CA type: MT4099    Number of ports: 2    Firmware version: 2.40.7000    Hardware version: 1    Node GUID: 0x7cfe90030016c230    System image GUID: 0x7cfe90030016c233    Port 1:        State: Active        Physical state: LinkUp        Rate: 56        Base lid: 3        LMC: 0        SM lid: 4        Capability mask: 0x02514868        Port GUID: 0x7cfe90030016c231        Link layer: InfiniBand    Port 2:        State: Down        Physical state: Polling        Rate: 10        Base lid: 0        LMC: 0        SM lid: 0        Capability mask: 0x02514868        Port GUID: 0x7cfe90030016c232        Link layer: InfiniBandCA 'mlx4_1'    CA type: MT4099    Number of ports: 2    Firmware version: 2.40.7000    Hardware version: 1    Node GUID: 0x7cfe900300917da0    System image GUID: 0x7cfe900300917da3    Port 1:        State: Active        Physical state: LinkUp        Rate: 56        Base lid: 8        LMC: 0        SM lid: 8        Capability mask: 0x0251486a        Port GUID: 0x7cfe900300917da1        Link layer: InfiniBand    Port 2:        State: Down        Physical state: Polling        Rate: 10        Base lid: 0        LMC: 0        SM lid: 0        Capability mask: 0x02514868        Port GUID: 0x7cfe900300917da2        Link layer: InfiniBand[root@host1 ~]# ibstatusInfiniband device 'mlx4_0' port 1 status:    default gid:     fe80:0000:0000:0000:7cfe:9003:0016:c231    base lid:     0x3    sm lid:         0x4    state:         4: ACTIVE    phys state:     5: LinkUp    rate:         56 Gb/sec (4X FDR)    link_layer:     InfiniBandInfiniband device 'mlx4_0' port 2 status:    default gid:     fe80:0000:0000:0000:7cfe:9003:0016:c232    base lid:     0x0    sm lid:         0x0    state:         1: DOWN    phys state:     2: Polling    rate:         10 Gb/sec (4X)    link_layer:     InfiniBandInfiniband device 'mlx4_1' port 1 status:    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7da1    base lid:     0x8    sm lid:         0x8    state:         4: ACTIVE    phys state:     5: LinkUp    rate:         56 Gb/sec (4X FDR)    link_layer:     InfiniBandInfiniband device 'mlx4_1' port 2 status:    default gid:     fe80:0000:0000:0000:7cfe:9003:0091:7da2    base lid:     0x0    sm lid:         0x0    state:         1: DOWN    phys state:     2: Polling    rate:         10 Gb/sec (4X)    link_layer:     InfiniBand[root@host1 ~]# ibv_devinfohca_id:    mlx4_1    transport:            InfiniBand (0)    fw_ver:                2.40.7000    node_guid:            7cfe:9003:0091:7da0    sys_image_guid:            7cfe:9003:0091:7da3    vendor_id:            0x02c9    vendor_part_id:            4099    hw_ver:                0x1    board_id:            MT_1090120019    phys_port_cnt:            2    Device ports:        port:    1            state:            PORT_ACTIVE (4)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            8            port_lid:        8            port_lmc:        0x00            link_layer:        InfiniBand        port:    2            state:            PORT_DOWN (1)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            0            port_lid:        0            port_lmc:        0x00            link_layer:        InfiniBandhca_id:    mlx4_0    transport:            InfiniBand (0)    fw_ver:                2.40.7000    node_guid:            7cfe:9003:0016:c230    sys_image_guid:            7cfe:9003:0016:c233    vendor_id:            0x02c9    vendor_part_id:            4099    hw_ver:                0x1    board_id:            MT_1090120019    phys_port_cnt:            2    Device ports:        port:    1            state:            PORT_ACTIVE (4)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            4            port_lid:        3            port_lmc:        0x00            link_layer:        InfiniBand        port:    2            state:            PORT_DOWN (1)            max_mtu:        4096 (5)            active_mtu:        4096 (5)            sm_lid:            0            port_lid:        0            port_lmc:        0x00            link_layer:        InfiniBand[root@host1 ~]# ibnetdiscover## Topology file: generated on Tue Jun  6 08:38:13 2017## Initiated from node 7cfe90030016c230 port 7cfe90030016c231vendid=0x2c9devid=0xc738sysimgguid=0x7cfe900300f17940switchguid=0x7cfe900300f17940(7cfe900300f17940)Switch    36 "S-7cfe900300f17940"        # "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0[31]    "H-7cfe90030016c230"[1](7cfe90030016c231)         # "host1 HCA-1" lid 3 4xFDR[33]    "H-248a0703005d07f0"[1](248a0703005d07f1)         # "rac2 HCA-1" lid 0 4xFDR[35]    "H-7cfe900300917f10"[1](7cfe900300917f11)         # "rac1 HCA-1" lid 2 4xFDRvendid=0x2c9devid=0x1003sysimgguid=0x7cfe900300917f13caguid=0x7cfe900300917f10Ca    2 "H-7cfe900300917f10"        # "rac1 HCA-1"[1](7cfe900300917f11)     "S-7cfe900300f17940"[35]        # lid 2 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDRvendid=0x2c9devid=0x1003sysimgguid=0x248a0703005d07f3caguid=0x248a0703005d07f0Ca    2 "H-248a0703005d07f0"        # "rac2 HCA-1"[1](248a0703005d07f1)     "S-7cfe900300f17940"[33]        # lid 0 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDRvendid=0x2c9devid=0x1003sysimgguid=0x7cfe90030016c233caguid=0x7cfe90030016c230Ca    2 "H-7cfe90030016c230"        # "host1 HCA-1"[1](7cfe90030016c231)     "S-7cfe900300f17940"[31]        # lid 3 lmc 0 "MF0;switch-138276:SX6036/U1" lid 1 4xFDR[root@host1 ~]# iblinkinfoCA: rac1 HCA-1:      0x7cfe900300917f11      2    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1   35[  ] "MF0;switch-138276:SX6036/U1" ( )CA: rac2 HCA-1:      0x248a0703005d07f1      0    1[  ] ==( 4X       14.0625 Gbps Initialize/  LinkUp)==>       1   33[  ] "MF0;switch-138276:SX6036/U1" ( )Switch: 0x7cfe900300f17940 MF0;switch-138276:SX6036/U1:           1    1[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    2[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    3[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    4[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1    9[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   10[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   11[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   12[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   13[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   14[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   15[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   16[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   17[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   18[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   19[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   20[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   21[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   22[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   23[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   24[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   25[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   26[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   27[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   28[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   29[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   30[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   31[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       3    1[  ] "host1 HCA-1" ( )           1   32[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   33[  ] ==( 4X       14.0625 Gbps Initialize/  LinkUp)==>       0    1[  ] "rac2 HCA-1" ( )           1   34[  ] ==(                Down/ Polling)==>             [  ] "" ( )           1   35[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       2    1[  ] "rac1 HCA-1" ( )           1   36[  ] ==(                Down/ Polling)==>             [  ] "" ( )CA: host1 HCA-1:      0x7cfe90030016c231      3    1[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>       1   31[  ] "MF0;switch-138276:SX6036/U1" ( )[root@host1 ~]# ibswitchesSwitch    : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0[root@host1 ~]# ibnodesCa    : 0x7cfe900300917f10 ports 2 "rac1 HCA-1"Ca    : 0x248a0703005d07f0 ports 2 "rac2 HCA-1"Ca    : 0x7cfe90030016c230 ports 2 "host1 HCA-1"Switch    : 0x7cfe900300f17940 ports 36 "MF0;switch-138276:SX6036/U1" enhanced port 0 lid 1 lmc 0[root@host1 ~]#