容器跨主机通信之flannel

VM1: 129.107.126.123
VM2: 129.107.126.142


1, install etcd and execute it

on VM1
$ etcd    -initial-advertise-peer-urls http://VM1-IP:2380 \
        -listen-peer-urls http://VM1-IP:2380 \
        -listen-client-urls http://VM1-IP:2379,http://127.0.0.1:4001 \
        -advertise-client-urls http://VM1-IP:2379 \
        -initial-cluster-token etcd-cluster \
        -initial-cluster tonysuo-VM1=http://VM1-IP:2380,tonysuo-VM2=http://VM2-IP:2380 \
        -initial-cluster-state new

on VM2 do the same thing, using different IP


2,  configure etcdctl

on VM1:

这里10.1.0.0是VM1上container的子网
$ etcdctl set /coreos.com/network/config '{"Network":"10.1.0.0/16", "SubnetLen":24, "SubnetMin":"10.1.1.0", "SubnetMax":"10.1.1.255", "Backend":{"Type":"vxlan" }}'

on VM2:

这里10.2.0.0是VM1上container的子网

$ etcdctl set /coreos.com/network/config '{"Network":"10.1.0.0/16", "SubnetLen":24, "SubnetMin":"10.1.2.0", "SubnetMax":"10.1.2.255", "Backend":{"Type":"vxlan" }}'

---------------------------------【说明】---------------------------------
这里是使用VXLAN模式，使用UDP模式如下：

$ etcdctl set /coreos.com/network/config 

'{"Network":"10.1.0.0/16", 

"SubnetLen":24, 

"SubnetMin":"10.1.1.0", 

"SubnetMax":"10.1.1.255", 

"Backend":{"Type":"udp", "Port":7890 }

}'

on vm1:
$ etcdctl set /coreos.com/network/subnets/10.1.2.0-24 '{"PublicIP":"129.107.126.220","BackendType":"udp","BackendData":{"VtepMAC":"26:c9:f5:71:9d:0c"}}'

这里publicIP是vm2的ip，VtepMAC是vm2的flannel的MAC

on vm2:
$ etcdctl set /coreos.com/network/subnets/10.1.1.0-24 '{"PublicIP":"129.107.126.193","BackendType":"udp","BackendData":{"VtepMAC":"d2:6b:a9:93:f4:2f"}}'

这里publicIP是vm1的ip，VtepMAC是vm1的flannel的MAC


---------------------------------------------------------------------------------

[Error] Failed to acquire subnet: out of subnets
[Solution] delete the subnet under /coreos.com/network/subnets, as follows:

$ etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/10.1.1.0-24
/coreos.com/network/subnets/10.1.2.0-24

$ etcdctl rm /coreos.com/network/subnets/10.1.1.0-24
PrevNode.Value: {"PublicIP":"129.107.126.123","BackendType":"vxlan","BackendData":{"VtepMAC":"ae:97:d1:7e:8c:75"}}

$ etcdctl rm /coreos.com/network/subnets/10.1.2.0-24
PrevNode.Value: {"PublicIP":"129.107.126.142","BackendType":"vxlan","BackendData":{"VtepMAC":"be:b7:aa:ed:1b:df"}}

add routing rule on etcd for visit remote container
on VM1, add ip forwording rules for visiting container on VM2


VM1:
ip: 129.107.126.123
MAC: 52:54:00:84:79:e0
VM2:
ip: 129.107.126.142
MAC: 52:54:00:ef:ee:25


on VM1, VM2:

这里public ip是虚拟机的ip，VtepMAC是虚拟机上的flannel.1端口的MAC地址
$ etcdctl set /coreos.com/network/subnets/10.1.1.0-24 '{"PublicIP":"129.107.126.123","BackendType":"vxlan","BackendData":{"VtepMAC":"ae:97:d1:7e:8c:75"}}'

$ etcdctl set /coreos.com/network/subnets/10.1.2.0-24 '{"PublicIP":"129.107.126.142","BackendType":"vxlan","BackendData":{"VtepMAC":"be:b7:aa:ed:1b:df"}}'

这里踩了无数的坑

3, run flannel

$ sudo flanneld

4, 配置docker

config network for docker0 and restart docker daemon with Flannel network configuration, execute commands as follows:

$ service docker stop

$ source /run/flannel/subnet.env

$ sudo ifconfig docker0 ${FLANNEL_SUBNET}

$ sudo docker daemon --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} &

5，run container

on VM1：
# docker run -t -i --name test-flannel-1  ubuntu:16.10 /bin/bash -P
# apt update -y
# apt install wget iputils-ping fish gcc openssh-client net-tools g++ make vim git autoconf automake libtool -y


on VM2：
# docker run -t -i --name test-flannel-2  ubuntu:16.10 /bin/bash -P
# apt update -y
# apt install wget iputils-ping fish gcc openssh-client net-tools g++ make vim git autoconf automake libtool -y




在container1上ping container2，可以ping通。

【bug说明】
bug 1：
L3 miss: <Service's IP>
Route for <Service's IP> not found

原因可能是etcdctl没有配置好，rule没加进来。确保每个vm上面的rule是一致的。check like this：

bug 2：
I0309 17:31:08.274836 11234 vxlan.go:340] Ignoring not a miss: 52:54:00:ef:ee:25, 10.1.2.2
I0309 17:31:11.346637 11234 vxlan.go:340] Ignoring not a miss: 52:54:00:ef:ee:25, 10.1.2.2
I0309 17:31:12.370653 11234 vxlan.go:340] Ignoring not a miss: 52:54:00:ef:ee:25, 10.1.2.2
I0309 17:31:13.394619 11234 vxlan.go:340] Ignoring not a miss: 52:54:00:ef:ee:25, 10.1.2.2

Ignoring not a miss的原因可能是public ip或者vtepmac没有配置正确。解决方法参加上文step 2。