部署kubernetes集群

在本例中使用三台机器来部署kubernetes集群

172.16.36.50    master172.16.36.51    cti-1172.16.36.54    cti-4

关闭三台机器的防火墙

systemctl stop firewalldsystemctl disable firewalld

分别修改三台机器的selinux配置文件

[root@cti-m kubernetes]# vi /etc/sysconfig/selinux==# This file controls the state of SELinux on the system.====# SELINUX= can take one of these three values:==# enforcing - SELinux security policy is enforced.====# permissive - SELinux prints warnings instead of enforcing.====# disabled - No SELinux policy is loaded.====#SELINUX=enforcing==SELINUX=disabled==# SELINUXTYPE= can take one of three two values:====# targeted - Targeted processes are protected,====# minimum - Modification of targeted policy. Only selected processes are protected.====# mls - Multi Level Security protection.==SELINUXTYPE=targeted

在三台机器的hosts文件中分别加入以下内容

[root@cti-m kubernetes]# vi /etc/hosts172.16.36.50   master172.16.36.51   cti-1172.16.36.54   cti-4

一.安装配置master 1.安装kubernetes和etcd

yum -y install kubernetes etcd

2.编辑/etc/etcd/etcd.conf，确保etcd监听所有的ip地址，这里配置etcd的方法参见《etcd集群配置》

3.配置ServiceAccount和Secret 使用openssl工具在master服务器上创建证书和私钥相关的文件，分别执行以下命令：

[root@cti-m kubernetes]# mkdir /var/run/kubernetes[root@cti-m kubernetes]# cd /var/run/kubernetes[root@cti-m kubernetes]# openssl genrsa -out ca.key 2048[root@cti-m kubernetes]# openssl req -x509 -new -nodes -key ca.key -subj "/CN=wecloud.com" -days 5000 -out ca.crt[root@cti-m kubernetes]# openssl genrsa -out server.key 2048[root@cti-m kubernetes]# openssl req -new -key server.key -subj "/CN=cti-m" -out server.csr[root@cti-m kubernetes]# openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 5000

注意：在生成server.csr时-subj参数中/CN指定的名字为master的主机名。另外，在生成ca.crt时-subj参数中/CN的名字最好与主机名不同，设置为相同可能导致对普通master的https访问认证失败

执行完成后会生成6个文件：ca.crt、ca.key、ca.srl、server.crt、server.csr、server.key

4.配置kube-apiserver，编辑/etc/kubernetes/apiserver，需要修改的配置如下：

[root@cti-m kubernetes]# egrep -v "^#|^$" /etc/kubernetes/apiserverKUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"KUBE_API_PORT="--port=8080"KUBELET_PORT="--kubelet-port=10250"KUBE_ETCD_SERVERS="--etcd-servers=http://172.16.36.50:2379,http://172.16.36.51:2379,http://172.16.36.54:2379"KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"KUBE_API_ARGS="--client_ca_file=/var/run/kubernetes/ca.crt --tls-private-key-file=/var/run/kubernetes/server.key --tls-cert-file=/var/run/kubernetes/server.crt"

注意：如果之前在创建kubernetes集群时因为不需要安全认证而去掉ServiceAccount，在这里需要添加回来,如果重启虚拟机后，这些证书需要重新生成，不然apiserver不能启动

5.配置kube-controller-manager，编辑/etc/kubernetes/controller-manager，需要修改的配置如下：

[root@cti-m kubernetes]# egrep -v "^#|^$" /etc/kubernetes/controller-managerKUBE_CONTROLLER_MANAGER_ARGS="--service_account_private_key_file=/var/run/kubernetes/server.key --root_ca_file=/var/run/kubernetes/ca.crt"

6.启动etcd, kube-apiserver, kube-controller-manager和kube-scheduler

for SERVICE in etcd kube-apiserver kube-controller-manager kube-scheduler; do    systemctl restart $SERVICE      systemctl enable $SERVICEsystemctl status $SERVICE  done

在kube-apiserver服务成功启动后，系统会自动为每个命名空间创建一个ServiceAccount和一个Secret（包含一个ca.crt和一个token）

[root@cti-m kubernetes]# kubectl get serviceaccount --all-namespacesNAMESPACE   NAME      SECRETS   AGEdefault     default   1         1h[root@cti-m kubernetes]# kubectl get secrets --all-namespacesNAMESPACE   NAME                  TYPE                                  DATA      AGEdefault     default-token-fq3j8   kubernetes.io/service-account-token   3         1h[root@cti-m kubernetes]# kubectl describe secret default-token-fq3j8Name:           default-token-fq3j8Namespace:      defaultLabels:         <none>Annotations:    kubernetes.io/service-account.name=default,kubernetes.io/service-account.uid=4c27b13e-ad51-11e6-a4d0-000c298207a9Type:   kubernetes.io/service-account-tokenData====token:          eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tZnEzajgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjRjMjdiMTNlLWFkNTEtMTFlNi1hNGQwLTAwMGMyOTgyMDdhOSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.mKTF9y7kUzWomc2GBwUvvQ0vZZvbQ0ojH_1NBzzOqH4kaYHE545xkpRqeHxCq91h19aXVMkT96mkxLsn75mDPAMoxUt238YqYrUTvsDZJz8NYknXZ18AMfylJYQsLi_6KO4aE1z8hDh-5R-y5jKhQoAOnNmK8uJKRfGoDLweHHqYCCeNPH-hAqdh7eisIvjpsFgUFsBtCJPrwoNVRboMZSqjItE2YEd_y0sgWxAoK1SQqg2JN3zOY3l2RHHj9y48FEDWI5Cf3nY4CTEqv5n97iggnNTi9JhGEEOkK9ZockvqAYMv4luVDqmCud2nZmuVV26Igdyp6IiHvC6WX8jvFQca.crt:         1099 bytesnamespace:      7 bytes

之后ReplicationController在创建Pod时，会生成类型为Secret的volume存储卷，并将该volume挂载到Pod内的如下目录中：/var/run/secrets/kubernetes.io/serviceaccount。然后，容器内的应用程序就可以使用该Secret与master建立https连接了。Pod的volume设置和挂载操作由ReplicationController和kubelet自动完成，可以通过查看Pod的详细信息了解到

7.在etcd里配置flannel网络

[root@cti-m kubernetes]#  etcdctl mk /flannel/network/config '{"Network":"172.17.0.0/16","SubnetMin":"172.17.1.0","SubnetMax":"172.17.254.0"}'[root@cti-m kubernetes]# etcdctl ls /flannel --recursive/flannel/network/flannel/network/config[root@cti-m kubernetes]# etcdctl get /flannel/network/config{"Network":"172.17.0.0/16","SubnetMin":"172.17.1.0","SubnetMax":"172.17.254.0"}

二.安装配置minion 1.安装kubernetes和flannel

yum -y install flannel kubernetes

2.为flannel配置etcd服务，编辑/etc/sysconfig/flanneld，修改如下内容

[root@cti-1 run]# egrep -v "^#|^$" /etc/sysconfig/flanneldFLANNEL_ETCD="http://172.16.36.50:2379,http://172.16.36.51:2379,http://172.16.36.54:2379"FLANNEL_ETCD_KEY="/flannel/network"

3.编辑kubernetes的全局配置文件/etc/kubernetes/config，修改如下内容：

[root@cti-1 run]# egrep -v "^#|^$" /etc/kubernetes/configKUBE_LOGTOSTDERR="--logtostderr=true"KUBE_LOG_LEVEL="--v=0"KUBE_ALLOW_PRIV="--allow-privileged=false"KUBE_MASTER="--master=http://master:8080"

4.编辑/etc/kubernetes/kubelet，修改如下内容：

[root@cti-1 run]# egrep -v "^#|^$" /etc/kubernetes/kubeletKUBELET_ADDRESS="--address=0.0.0.0"KUBELET_PORT="--port=10250"KUBELET_HOSTNAME="--hostname-override=cti-1"KUBELET_API_SERVER="--api-servers=http://master:8080"KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=gcr.io/google_containers/pause:0.8.0"KUBELET_ARGS=""

5.修改docker的配置文件/etc/sysconfig/docker（vi /usr/lib/systemd/system/docker.service），修改内容如下：

# /etc/sysconfig/docker# Modify these options if you want to change the way the docker daemon runs#OPTIONS='--selinux-enabled --log-driver=journald'OPTIONS='-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock'DOCKER_CERT_PATH=/etc/docker

如果还是不能启动docker engine并且显示是如下错误：

Job for docker.service failed because the control process exited with error codeCreated symlink from /etc/systemd/system/multi-user.target.wants/docker.service● docker.service - Docker Application Container Engine   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor prese  Drop-In: /usr/lib/systemd/system/docker.service.d           └─flannel.conf   Active: failed (Result: exit-code) since 五 2016-11-18 22:32:26 CST; 101ms ag     Docs: http://docs.docker.com Main PID: 22472 (code=exited, status=1/FAILURE)11月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.8223189311月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.8223386111月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.8623824911月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.8624271511月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.8624519511月 18 22:32:26 cti-4 docker-current[22472]: time="2016-11-18T22:32:26.8847918811月 18 22:32:26 cti-4 systemd[1]: docker.service: main process exited, code=exi11月 18 22:32:26 cti-4 systemd[1]: Failed to start Docker Application Container11月 18 22:32:26 cti-4 systemd[1]: Unit docker.service entered failed state.11月 18 22:32:26 cti-4 systemd[1]: docker.service failed.Hint: Some lines were ellipsized, use -l to show in full.

解决方法：removing /var/lib/docker/network

6.启动flanneld, kube-proxy, kubelet和docker服务

for SERVICE in flanneld kube-proxy kubelet docker; do      systemctl restart $SERVICE      systemctl enable $SERVICE      systemctl status $SERVICE  done

7.在master上查看节点信息：

[root@cti-m kubernetes]# kubectl get nodeNAME      STATUS    AGEcti-1     Ready     1hcti-4     Ready     1h

8.判断节点是否能连接master：

[root@cti-4 ~]# curl -s -L http://172.16.36.50:2379/version{"etcdserver":"2.3.7","etcdcluster":"2.3.0"}[root@cti-4 ~]#

9.在master节点查看flannel子网分配情况：

[root@cti-m kubernetes]# etcdctl ls /flannel/network/subnets/flannel/network/subnets/172.17.58.0-24/flannel/network/subnets/172.17.70.0-24