《kubernetes-1.8.0》02-etcd群集搭建

—《kubernetes 1.8.0 测试环境安装部署》

— 时间：2017-11-22

一、etcd软件安装

通过在node-131、node-132、node-133上yum安装，也可使用mritd提供tarball中的etcd rpm包。

yum install -y etcd 

当前版本为”etcd-3.2.701.el7”

二、分发etcd相关证书

分发证书

$ cd ~/etcd_ssl$ for IP in `seq 131 133`;do    ssh root@172.18.169.$IP mkdir /etc/etcd/ssl    scp *.pem root@172.18.169.$IP:/etc/etcd/ssl    ssh root@172.18.169.$IP chown -R etcd:etcd /etc/etcd/ssl    ssh root@172.18.169.$IP chmod -R 644 /etc/etcd/ssl/*    ssh root@172.18.169.$IP chmod 755 /etc/etcd/ssldone

设置etcd数据目录owner/group

for IP in `seq 131 133`;do    ssh root@172.18.169.$IP chown -R etcd:etcd /var/lib/etcddone

三、修改etcd配置文件

$ vim /etc/etcd/etcd.conf

示例：

# [member]ETCD_NAME=node-131ETCD_DATA_DIR="/var/lib/etcd/node-131.etcd"ETCD_WAL_DIR="/var/lib/etcd/wal"ETCD_SNAPSHOT_COUNT="100"ETCD_HEARTBEAT_INTERVAL="100"ETCD_ELECTION_TIMEOUT="1000"ETCD_LISTEN_PEER_URLS="https://172.18.169.131:2380"ETCD_LISTEN_CLIENT_URLS="https://172.18.169.131:2379,http://127.0.0.1:2379"ETCD_MAX_SNAPSHOTS="5"ETCD_MAX_WALS="5"#ETCD_CORS=""# [cluster]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.18.169.131:2380"# if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."ETCD_INITIAL_CLUSTER="node-131=https://172.18.169.131:2380,node-132=https://172.18.169.132:2380,node-133=https://172.18.169.133:2380"ETCD_INITIAL_CLUSTER_STATE="new"ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"ETCD_ADVERTISE_CLIENT_URLS="https://172.18.169.131:2379"#ETCD_DISCOVERY=""#ETCD_DISCOVERY_SRV=""#ETCD_DISCOVERY_FALLBACK="proxy"#ETCD_DISCOVERY_PROXY=""#ETCD_STRICT_RECONFIG_CHECK="false"#ETCD_AUTO_COMPACTION_RETENTION="0"# [proxy]#ETCD_PROXY="off"#ETCD_PROXY_FAILURE_WAIT="5000"#ETCD_PROXY_REFRESH_INTERVAL="30000"#ETCD_PROXY_DIAL_TIMEOUT="1000"#ETCD_PROXY_WRITE_TIMEOUT="5000"#ETCD_PROXY_READ_TIMEOUT="0"# [security]ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"ETCD_CLIENT_CERT_AUTH="true"ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"ETCD_AUTO_TLS="true"ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"ETCD_PEER_CLIENT_CERT_AUTH="true"ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"ETCD_PEER_AUTO_TLS="true"# [logging]#ETCD_DEBUG="false"# examples for -log-package-levels etcdserver=WARNING,security=DEBUG#ETCD_LOG_PACKAGE_LEVELS=""

• ETCD_NAME: etcd节点名称，如果是静态etcd cluster，必须与ETCD_INITIAL_CLUSTER中的名称进行对应。
• ETCD_INITIAL_CLUSTER_STATE: new为新建集群，如果是加入一个已经存在的etcd集群，需将该参数改为existing
• ETCD_DATA_DIR=：存放etcd member等db数据
• ETCD_CLIENT_CERT_AUTH、ETCD_TRUSTED_CA_FILE、ETCD_CERT_FILE、ETCD_KEY_FILE等：为etcd TLS所需证书，制定之前创建的证书即可。
• 更多etcd TLS 选项查看 etcd 3.2.7 document

node-132、node-133修改对应参数。

三、启动etcd服务并验证

node-131、node-132、node-133上执行：

systemctl daemon-reloadsystemctl start etcdsystemctl enable etcd

检查节点状态：

$ export ETCDCTL_API=3$ etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \--cert=/etc/etcd/ssl/etcd.pem \--key=/etc/etcd/ssl/etcd-key.pem \--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \endpoint healthhttps://172.18.169.133:2379 is healthy: successfully committed proposal: took = 2.016793mshttps://172.18.169.132:2379 is healthy: successfully committed proposal: took = 2.005839mshttps://172.18.169.131:2379 is healthy: successfully committed proposal: took = 1.167565ms

检查etcd 版本：

[root@node-131 etcd_ssl]# etcdctl versionetcdctl version: 3.2.7API version: 3.2

加上TLS之后 etcd api verison 自动切换成了3.2。原先的cluster-health ls pwd 什么的都不管用了。

四、删除、添加etcd节点

查看群集成员：

$ export ETCDCTL_API=3$ etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \--cert=/etc/etcd/ssl/etcd.pem \--key=/etc/etcd/ssl/etcd-key.pem \--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \member list5d5554b1f11aba62, started, node-131, https://172.18.169.131:2380, https://172.18.169.131:23798b10a60fc4b98fcb, started, node-133, https://172.18.169.133:2380, https://172.18.169.133:2379cd1bf9a8ae65b314, started, node-132, https://172.18.169.132:2380, https://172.18.169.132:2379

删除note-133：

$ etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \--cert=/etc/etcd/ssl/etcd.pem \--key=/etc/etcd/ssl/etcd-key.pem \--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \member remove 8b10a60fc4b98fcbMember 8b10a60fc4b98fcb removed from cluster 3697c33650b7b984$ etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \--cert=/etc/etcd/ssl/etcd.pem \--key=/etc/etcd/ssl/etcd-key.pem \--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \member list5d5554b1f11aba62, started, node-131, https://172.18.169.131:2380, https://172.18.169.131:2379cd1bf9a8ae65b314, started, node-132, https://172.18.169.132:2380, https://172.18.169.132:2379

将node-133添加回etcd群集：

群集节点添加member：

etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \--cert=/etc/etcd/ssl/etcd.pem \--key=/etc/etcd/ssl/etcd-key.pem \--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \member add node-133 \--peer-urls=https://172.18.169.133:2380 Member 17948fc49f73cbb9 added to cluster 3697c33650b7b984ETCD_NAME="node-133"ETCD_INITIAL_CLUSTER="node-133=https://172.18.169.133:2380,node-131=https://172.18.169.131:2380,node-132=https://172.18.169.132:2380"ETCD_INITIAL_CLUSTER_STATE="existing"

• --peer-urls：api 3.2后加member需要增加这个选项
  修改member设置

##清空member上的数据目录$ sudo systemctl start etcd$ sudo rm -rf /var/lib/etcd/*##修改member etcd配置文件修改ETCD_INITIAL_CLUSTER_STATE="existing"##启动服务$ sudo systemctl start etcd

验证节点是否加入：

$ sudo etcdctl \--cacert=/etc/etcd/ssl/etcd-root-ca.pem \--cert=/etc/etcd/ssl/etcd.pem \--key=/etc/etcd/ssl/etcd-key.pem \--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \member list17948fc49f73cbb9, started, node-133, https://172.18.169.133:2380, https://172.18.169.133:23795d5554b1f11aba62, started, node-131, https://172.18.169.131:2380, https://172.18.169.131:2379cd1bf9a8ae65b314, started, node-132, https://172.18.169.132:2380, https://172.18.169.132:2379$ sudo etcdctl \--cacert=/etc/etcd/ssl/etcd-root-ca.pem \--cert=/etc/etcd/ssl/etcd.pem \--key=/etc/etcd/ssl/etcd-key.pem \--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \endpoint healthhttps://172.18.169.133:2379 is healthy: successfully committed proposal: took = 2.348909mshttps://172.18.169.132:2379 is healthy: successfully committed proposal: took = 2.139596mshttps://172.18.169.131:2379 is healthy: successfully committed proposal: took = 1.222221ms

至此etcd TLS集群搭建完成

本系列其他内容：

• 01-环境准备

• 02-etcd群集搭建

• 03-kubectl管理工具

• 04-master搭建

• 05-node节点搭建

• 06-addon-calico

• 07-addon-kubedns

• 08-addon-dashboard

• 09-addon-kube-prometheus

• 10-addon-EFK

• 11-addon-Harbor

• 12-addon-ingress-nginx

• 13-addon-traefik

参考链接：

https://mritd.me/2017/10/09/set-up-kubernetes-1.8-ha-cluster/

https://github.com/opsnull/follow-me-install-kubernetes-cluster

https://coreos.com/etcd/docs/3.2.7/index.html