《kubernetes-1.8.0》06-addon-calico

《kubernetes 1.8.0 测试环境安装部署》

时间：2017-11-23

一、修改calico配置

在mritd.me/部署 Calico中提及：

官方文档中直接创建的 calico.yml 文件中，使用 DaemonSet 方式启动 calico-node，同时 calico-node 的 IP 设置和 NODENAME 设置均为空，此时 calico-node 会进行自动获取，网络复杂情况下获取会出现问题；比如 IP 拿到了 docker 网桥的 IP，NODENAME 获取不正确等，最终导致出现很奇怪的错误

经测试 2.6.1 calico-node镜像版本确实有这样的问题，漠然的方法是calico node采用systemd的方式控制，其他组件通过daemonset安装。后续calico node镜像升级成 2.6.2 该问题就没有再出现。

获取最新的calico.yaml：

$sudo mkdir ~/calico/$cd ~/calico/$wget https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/calico.yaml

查看calico-node所采用的镜像版本：

修改calico.yaml文件：

# 替换 Etcd 地址sed -i 's@.*etcd_endpoints:.*@\ \ etcd_endpoints:\ \"https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379\"@gi' calico.yaml# 替换 Etcd 证书export ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'`export ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'`export ETCD_CA=`cat /etc/etcd/ssl/etcd-root-ca.pem | base64 | tr -d '\n'`sed -i "s@.*etcd-cert:.*@\ \ etcd-cert:\ ${ETCD_CERT}@gi" calico.yamlsed -i "s@.*etcd-key:.*@\ \ etcd-key:\ ${ETCD_KEY}@gi" calico.yamlsed -i "s@.*etcd-ca:.*@\ \ etcd-ca:\ ${ETCD_CA}@gi" calico.yamlsed -i 's@.*etcd_ca:.*@\ \ etcd_ca:\ "/calico-secrets/etcd-ca"@gi' calico.yamlsed -i 's@.*etcd_cert:.*@\ \ etcd_cert:\ "/calico-secrets/etcd-cert"@gi' calico.yamlsed -i 's@.*etcd_key:.*@\ \ etcd_key:\ "/calico-secrets/etcd-key"@gi' calico.yaml

二、修改kubelet配置

根据官方文档要求 kubelet 配置必须增加--network-plugin=cni选项，所以需要修改 kubelet 配置:

#### kubernetes kubelet (minion) config# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)KUBELET_ADDRESS="--address=172.18.169.131"# The port for the info server to serve on# KUBELET_PORT="--port=10250"# You may leave this blank to use the actual hostnameKUBELET_HOSTNAME="--hostname-override=node.131"# location of the api-server# KUBELET_API_SERVER=""# Add your own!KUBELET_ARGS="--cgroup-driver=cgroupfs \              --network-plugin=cni \              --cluster-dns=10.254.0.2 \              --resolv-conf=/etc/resolv.conf \              --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \              --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \              --fail-swap-on=false \              --cert-dir=/etc/kubernetes/ssl \              --cluster-domain=cluster.local. \              --hairpin-mode=promiscuous-bridge \              --serialize-image-pulls=false \              --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0"

分别重启4个节点的kubelet：

systemctl daemon-reloadsystemctl restart kubelet

查看节点状态：

[root@node-131 calico]# kubectl get nodeNAME       STATUS     ROLES     AGE       VERSIONnode.131   NotReady   <none>    12h       v1.8.0node.132   NotReady   <none>    12h       v1.8.0node.133   NotReady   <none>    12h       v1.8.0node.134   NotReady   <none>    12h       v1.8.0

此时执行 kubectl get node 会看到 Node 为 NotReady 状态，属于正常情况

三、创建calico Daemonset

# 先创建 RBACkubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/rbac.yaml# 再创建 Calico Daemonsetkubectl create -f calico.yaml

quay.io仓库的镜像还是拖的动的，这里就不docker load了，除了calico-node image，其他的镜像可以通过mritd提供的tarball进行load。

检查Daemonset和相应pod运行情况：

[root@node-131 images]# kubectl get pods -n kube-systemNAME                                      READY     STATUS    RESTARTS   AGEcalico-kube-controllers-94b7cb897-krckw   1/1       Running   0          29mcalico-node-5dc8z                         2/2       Running   0          29mcalico-node-gm9k8                         2/2       Running   0          29mcalico-node-kt5fk                         2/2       Running   0          29mcalico-node-xds45                         2/2       Running   0          29m[root@node-131 images]# kubectl get ds -n kube-systemNAME          DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGEcalico-node   4         4         4         4            4           <none>          29m

重启kubelet、docker：

systemctl restart kubeletsystemctl restart docker

四、测试跨主机通讯

创建测试实例：

## 创建 deployment$ mkdir ~/demo$ cd ~/demo$ cat << EOF >> demo.deploy.ymlapiVersion: apps/v1beta2kind: Deploymentmetadata:  name: demo-deploymentspec:  replicas: 4  selector:    matchLabels:      app: demo  template:    metadata:      labels:        app: demo    spec:      containers:      - name: demo        image: mritd/demo        imagePullPolicy: IfNotPresent        ports:        - containerPort: 80EOF$ kubectl create -f demo.deploy.yml

验证通信：

[root@node-131 images]# kubectl get pod -o wideNAME                               READY     STATUS    RESTARTS   AGE       IP               NODEdemo-deployment-5fc9c54fb4-5pgfk   1/1       Running   0          2m        192.168.177.65   node.132demo-deployment-5fc9c54fb4-5svgl   1/1       Running   0          2m        192.168.33.193   node.131demo-deployment-5fc9c54fb4-dfcfd   1/1       Running   0          2m        192.168.188.1    node.133demo-deployment-5fc9c54fb4-dttvb   1/1       Running   0          2m        192.168.56.65    node.134[root@node-131 images]# kubectl exec -ti demo-deployment-5fc9c54fb4-5svgl bashbash-4.3# ping 192.168.56.66PING 192.168.56.66 (192.168.56.66): 56 data bytes64 bytes from 192.168.56.66: seq=0 ttl=62 time=0.407 ms^C--- 192.168.56.66 ping statistics ---1 packets transmitted, 1 packets received, 0% packet lossround-trip min/avg/max = 0.407/0.407/0.407 ms

至此，群集网络组件calico搭建完成

本系列其他内容：

• 01-环境准备

• 02-etcd群集搭建

• 03-kubectl管理工具

• 04-master搭建

• 05-node节点搭建

• 06-addon-calico

• 07-addon-kubedns

• 08-addon-dashboard

• 09-addon-kube-prometheus

• 10-addon-EFK

• 11-addon-Harbor

• 12-addon-ingress-nginx

• 13-addon-traefik

参考链接：

https://mritd.me/2017/10/09/set-up-kubernetes-1.8-ha-cluster/

https://docs.projectcalico.org/v2.6/getting-started/kubernetes/