XX省移动CMNET三期扩容项目遇到IPMP BUG解决过程

1.故障现象

XX移动网管中心CMNET三期扩容T5120*4+ST2530*1安装。安装系统完成后配置probe-based IPMP后，首次网卡切换测试，拔出e1000g0网线后，浮动IP自动切换到e1000g1，再插入e1000g0网线后,浮动IP无法切回，e1000g0网卡显示failed,重启系统后，系统始终检测到第二块网卡failed，导致ipmp无法切换。

 

Jul 29 16:31:42 HB01-DNS-SV02 in.mpathd[249]: NIC failure detected on e1000g1 of group sc_ipmp0

Jul 29 16:31:42 HB01-DNS-SV02 in.mpathd[240]: Successfully failed over from NIC e1000g1 to NIC e1000g0

root@HB01-DNS-SV02 # ifconfig -a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1

        inet 127.0.0.1 netmask ff000000

e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

        inet 192.168.100.6 netmask ffffff00 broadcast 192.168.100.255

        groupname sc_ipmp0

        ether 0:21:28:3a:fa:36

e1000g0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2

        inet 192.168.100.5 netmask ffffff00 broadcast 192.168.100.255

e1000g1: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 3

        inet 192.168.100.4 netmask ffffff00 broadcast 192.168.100.255

        groupname sc_ipmp0

bash-3.00# more /etc/host*

/etc/hostname.e1000g0

::::::::::::::

HB01-DNS-SV02 netmask + broadcast + group sc_ipmp0 up /

addif HB01-DNS-SV02-e1000g0-test netmask + broadcast + deprecated -failover up

::::::::::::::

/etc/hostname.e1000g1

::::::::::::::

HB01-DNS-SV02-e1000g1-test netmask + broadcast + group sc_ipmp0 deprecated -failover up

::::::::::::::

/etc/hosts

::::::::::::::

#

# Internet host table

#

::1     localhost      

127.0.0.1       localhost      

192.168.100.6   HB01-DNS-SV02   HB01-DNS-SV02.  loghost

192.168.100.4   HB01-DNS-SV02-e1000g0-test     

192.168.100.5   HB01-DNS-SV02-e1000g1-test

 

2 解决过程

2.1排除硬件故障

2.1.1排除网卡故障

检查local-mac-address?=true

使用e1000g2和e1000g3配置ipmp后，故障依旧,系统也检测到e1000g3为failed,同时另外3台T5120也有同样的问题。因此排除了主机网卡故障

2.1.2排除网关故障

   找用户要了两台cisco2950 switch和dplink hub，将主机两个网卡连接switch或hub上，自己笔记本设置网关，然后做切换测试，问题依然存在，排除了网关故障

 分析有可能是ipmp的配置或者有bug导致

2.2解决方案

2.2.1配置基于link-based IPMP

之前配置probe-based IPMP基于IP的OSI模型第三层网络层,需ping通网关.在solairs10之后推出了link-base d IPMP基于OSI模型第二层链路层，无需测试IP，也无需ping通网关.建议大家采用link-based IPMP，即可节约IP资源，也可省去很多麻烦事。

 

root@HB01-DNS-SV02# more /etc/host*

::::::::::::::

/etc/hostname.e1000g0

::::::::::::::

HB01-DNS-SV02 netmask + broadcast + group ipmp_group0 up

::::::::::::::

/etc/hostname.e1000g1

::::::::::::::

HB01-DNS-SV02-e1000g1-test netmask + broadcast + group ipmp_group0 up

::::::::::::::

/etc/hosts

::::::::::::::

#

# Internet host table

#

::1     localhost      

127.0.0.1       localhost      

192.168.100.6   HB01-DNS-SV02   HB01-DNS-SV02.  loghost

#192.168.100.4   HB01-DNS-SV02-e1000g0-test     

192.168.100.5   HB01-DNS-SV02-e1000g1-test

root@HB01-DNS-SV02 # ifconfig -a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1

        inet 127.0.0.1 netmask ff000000

e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

        inet 192.168.100.6 netmask ffffff00 broadcast 192.168.100.255

        groupname ipmp_group0

        ether 0:21:28:3a:fa:34

e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

        inet 192.168.100.5 netmask ffffff00 broadcast 192.168.100.255

        groupname ipmp_group0

        ether 0:21:28:3a:fa:35

2.2.2此为系统BUG，升级到142900-02以后

后经查资料发现此为solaris 10 u8的一个bug ，bug id为271519 Solaris 10 Kernel Patches 141444-09 and 141445-09 May CauseInterfaceFailure in IP Multipathing (IPMP)

经测试打EIS 2.2.4 2010.06可解决此故障。

3 总结

建议大家采用link-based IPMP，link-based IPMP支持以下网卡：

Solaris OS:

 hme

 eri

ce

 ge

bge

 qfe

dmfe