kubeadm快速部署kubernetes(HA)

当前版本的kubeadm原生并不支持部署HA模式集群，但是实际上可以使用kubeadm部署后，再进行少量手动修改，即可实现HA模式的kubernetes集群。本次部署基于Ubuntu16.04，并使用最新的docker版本：17.06，kubernetes适用1.7.x版本，本文采用1.7.6。
Kubernetes&Docker技术交流QQ群：491137983，一起学习，共同进步！

1 环境准备

准备了六台机器作安装测试工作，机器信息如下:

IP Name Role OS 172.16.2.1 Master01 Controller,etcd Ubuntu16.04 172.16.2.2 Master02 Controller,etcd Ubuntu16.04 172.16.2.3 Master03 Controller,etcd Ubuntu16.04 172.16.2.11 Node01 Compute Ubuntu16.04 172.16.2.12 Node02 Compute Ubuntu16.04 172.16.2.13 Node03 Compute Ubuntu16.04 172.16.2.100 VIP VIP -

2 安装docker

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"apt-get update && apt-upgradeapt-get install docker-ce=17.06.0~ce-0~ubuntu

3 安装etcd集群

使用了docker-compose安装，当然，如果觉得麻烦，也可以直接docker run。

Master01节点的ETCD的docker-compose.yml：

etcd:  image: quay.io/coreos/etcd:v3.1.5  command: etcd --name etcd-srv1 --data-dir=/var/etcd/calico-data --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://172.16.2.1:2379,http://172.16.2.1:2380 --initial-advertise-peer-urls http://172.16.2.1:2380 --listen-peer-urls http://0.0.0.0:2380 -initial-cluster-token etcd-cluster -initial-cluster "etcd-srv1=http://172.16.2.1:2380,etcd-srv2=http://172.16.2.2:2380,etcd-srv3=http://172.16.2.3:2380" -initial-cluster-state new  net: "bridge"  ports:  - "2379:2379"  - "2380:2380"  restart: always  stdin_open: true  tty: true  volumes:  - /store/etcd:/var/etcd

Master02节点的ETCD的docker-compose.yml：

etcd:  image: quay.io/coreos/etcd:v3.1.5  command: etcd --name etcd-srv2 --data-dir=/var/etcd/calico-data --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://172.16.2.2:2379,http://172.16.2.2:2380 --initial-advertise-peer-urls http://172.16.2.2:2380 --listen-peer-urls http://0.0.0.0:2380 -initial-cluster-token etcd-cluster -initial-cluster "etcd-srv1=http://172.16.2.1:2380,etcd-srv2=http://172.16.2.2:2380,etcd-srv3=http://172.16.2.3:2380" -initial-cluster-state new  net: "bridge"  ports:  - "2379:2379"  - "2380:2380"  restart: always  stdin_open: true  tty: true  volumes:  - /store/etcd:/var/etcd

Master03节点的ETCD的docker-compose.yml：

etcd:  image: quay.io/coreos/etcd:v3.1.5  command: etcd --name etcd-srv3 --data-dir=/var/etcd/calico-data --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://172.16.2.3:2379,http://172.16.2.3:2380 --initial-advertise-peer-urls http://172.16.2.3:2380 --listen-peer-urls http://0.0.0.0:2380 -initial-cluster-token etcd-cluster -initial-cluster "etcd-srv1=http://172.16.2.1:2380,etcd-srv2=http://172.16.2.2:2380,etcd-srv3=http://172.16.2.3:2380" -initial-cluster-state new  net: "bridge"  ports:  - "2379:2379"  - "2380:2380"  restart: always  stdin_open: true  tty: true  volumes:  - /store/etcd:/var/etcd

创建好docker-compose.yml文件后，使用命令docker-compose up -d部署。

关于docker-compose的使用，可以参考：docker-compose安装文档。

3 安装k8s工具包

三种方式：博主提供、官方源安装和release工程编译，apt-get方式因为不能直接使用google提供的源，非官方源中提供的版本比较老，如果要使用新版本，可以尝试release工程编译的方式或者用博主提供的包下载。

博主提供

一些比较懒得同学:-D，可以直接从博主提供的位置下载RPM工具包安装，下载地址。

#安装kubelet的依赖包apt-get install -y socat ebtablesdpkg -i kubelet_1.7.6-00_amd64.deb kubeadm_1.7.6-00_amd64.deb kubernetes-cni_0.5.1-00_amd64.deb kubectl_1.7.6-00_amd64.deb

官方源安装

跨越GFW方式不细说，你懂的。

apt-get update && apt-get install -y apt-transport-httpscurl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -cat <<EOF >/etc/apt/sources.list.d/kubernetes.listdeb http://apt.kubernetes.io/ kubernetes-xenial mainEOFapt-get updateapt-get install -y kubelet kubeadm kubernetes-cni kubectl

默认安装最新的stable版本，可以根据需要指定安装版本apt-get install -y kubeadm=1.7.6-00，版本信息可以使用命令查看：apt-cache madison kubeadm。

relese编译

git clone https://github.com/kubernetes/release.gitdocker build --tag=debian-packager debiandocker run --volume="$(pwd)/debian:/src" debian-packager

编译完成后生成deb包到：debian/bin，进入到该目录后安装deb包。

4 镜像准备

4.1 下载docker镜像

kubeadm方式安装kubernetes集群需要的镜像在docker官方镜像中并未提供，只能去google的官方镜像库：gcr.io 中下载，GFW咋办？这里针对k8s-1.7.6在DockerHub上做了跳板镜像，各位可以直接下载，dashboard的版本并未紧跟kubelet主线版本，用哪个版本都可以，本文使用kubernetes-dashboard-amd64:v1.7.0。

kubernetes-1.7.6所需要的镜像：

• etcd-amd64:3.0.17
• pause-amd64:3.0
• kube-proxy-amd64:v1.7.6
• kube-scheduler-amd64:v1.7.6
• kube-controller-manager-amd64:v1.7.6
• kube-apiserver-amd64:v1.7.6
• kubernetes-dashboard-amd64:v1.7.0
• k8s-dns-sidecar-amd64:1.14.4
• k8s-dns-kube-dns-amd64:1.14.4
• k8s-dns-dnsmasq-nanny-amd64:1.14.4

偷下懒吧，直接执行以下脚本，提前下载好镜像，后边的动作就快了：

#!/bin/bashimages=(kube-proxy-amd64:v1.7.6 kube-scheduler-amd64:v1.7.6 kube-controller-manager-amd64:v1.7.6 kube-apiserver-amd64:v1.7.6 etcd-amd64:3.0.17 pause-amd64:3.0 kubernetes-dashboard-amd64:v1.6.1 k8s-dns-sidecar-amd64:1.14.4 k8s-dns-kube-dns-amd64:1.14.4 k8s-dns-dnsmasq-nanny-amd64:1.14.4)for imageName in ${images[@]} ; do  docker pull cloudnil/$imageNamedone

4.2 KUBE_REPO_PREFIX配置

通过KUBE_REPO_PREFIX配置官方镜像包的仓库位置，才可以直接使用从DockerHub上下载的镜像，请使用以下命令增加配置：1.KUBE_REPO_PREFIX环境变量 2.KUBELET_EXTRA_ARGS参数。

sed -i '/mesg n/i\export KUBE_REPO_PREFIX=cloudnil' ~/.profilesource ~/.profilecat > /etc/systemd/system/kubelet.service.d/20-extra-args.conf <<EOF[Service]Environment="KUBELET_EXTRA_ARGS=--pod-infra-container-image=cloudnil/pause-amd64:3.0"EOFsystemctl daemon-reloadsystemctl restart kubelet

5 安装master节点

由于kubeadm和kubelet安装过程中会生成/etc/kubernetes目录，而kubeadm init会先检测该目录是否存在，所以我们先使用kubeadm初始化环境。

kubeadm resetkubeadm init --api-advertise-addresses=172.16.2.1 --use-kubernetes-version v1.7.6

如果使用外部etcd集群，以前的kubeadm版本的--external-etcd-endpoints参数已经没有了，所以要使用–config参数外挂配置文件kubeadm-config.yml：

apiVersion: kubeadm.k8s.io/v1alpha1kind: MasterConfiguration# networking:#   podSubnet: 10.244.0.0/16apiServerCertSANs:- master01- master02- master03- 172.16.2.1- 172.16.2.2- 172.16.2.3- 172.16.2.100etcd:  endpoints:  - http://172.16.2.1:2379  - http://172.16.2.2:2379  - http://172.16.2.3:2379token: 67e411.zc3617bb21ad7ee3kubernetesVersion: v1.7.6

PS：token是使用指令kubeadm token generate生成的。

初始化指令：

kubeadm init --config kubeadm-config.yml

说明：如果打算使用flannel网络，请去掉networking注释。如果有多网卡的，请根据实际情况配置--api-advertise-addresses=<ip-address>，单网卡情况可以省略。

安装过程大概2-3分钟，输出结果如下：

[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.[preflight] Running pre-flight checks[init] Using Kubernetes version: v1.7.6[tokens] Generated token: "67e411.zc3617bb21ad7ee3"[certificates] Generated Certificate Authority key and certificate.[certificates] Generated API Server key and certificate[certificates] Generated Service Account signing keys[certificates] Created keys and certificates in "/etc/kubernetes/pki"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"[apiclient] Created API client, waiting for the control plane to become ready[apiclient] All control plane components are healthy after 21.317580 seconds[apiclient] Waiting for at least one node to register and become ready[apiclient] First node is ready after 6.556101 seconds[apiclient] Creating a test deployment[apiclient] Test deployment succeeded[addons] Created essential addon: kube-proxyYour Kubernetes master has initialized successfully!You should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:    http://kubernetes.io/docs/admin/addons/You can now join any number of machines by running the following on each node:kubeadm join --token=67e411.zc3617bb21ad7ee3 172.16.2.1

修改/etc/kubernetes/manifests/kube-apiserver.yaml中的admission-control策略：

root@master01:/etc/kubernetes/manifests# vi kube-apiserver.yaml#- --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota- --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds

复制/etc/kubernetes到Master02和Master03

scp -r /etc/kubernetes 172.16.2.2:/etc/kubernetesscp -r /etc/kubernetes 172.16.2.3:/etc/kubernetes

修改/etc/kubernetes和/etc/kubernetes/manifests以下文件：

#Master02root@master02:/etc/kubernetes# sed -i 's/172.16.2.1:6443/172.16.2.2:6443/g' `grep 172.16.2.1:6443 . -rl`root@master02:/etc/kubernetes# sed -i 's/--advertise-address=172.16.2.1/--advertise-address=172.16.2.2/g' manifests/kube-apiserver.yaml#Master03root@master03:/etc/kubernetes# sed -i 's/172.16.2.1:6443/172.16.2.3:6443/g' `grep 172.16.2.1:6443 . -rl`root@master03:/etc/kubernetes# sed -i 's/--advertise-address=172.16.2.1/--advertise-address=172.16.2.3/g' manifests/kube-apiserver.yaml

Master02和Master03节点重启kubelet服务。

systemctl restart kubelet

6 安装keepalived

Master01、Master02、Master03上分别安装keepalived，配置VIP为172.16.2.100。

详细过程略过，网络上很多教程，请各位自行查阅。

7 安装Node节点

Master节点安装好了Node节点就简单了。

kubeadm resetkubeadm join --token=67e411.zc3617bb21ad7ee3 172.16.2.100

输出结果如下：

[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.[preflight] Running pre-flight checks[preflight] Starting the kubelet service[tokens] Validating provided token[discovery] Created cluster info discovery client, requesting info from "http://172.16.2.100:9898/cluster-info/v1/?token-id=f11877"[discovery] Cluster info object received, verifying signature using given token[discovery] Cluster info signature and contents are valid, will use API endpoints [https://172.16.2.100:6443][bootstrap] Trying to connect to endpoint https://172.16.2.100:6443[bootstrap] Detected server version: v1.7.6[bootstrap] Successfully established connection with endpoint "https://172.16.2.100:6443"[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request[csr] Received signed certificate from the API server:Issuer: CN=kubernetes | Subject: CN=system:node:yournode | CA: falseNot before: 2017-06-28 19:44:00 +0000 UTC Not After: 2018-06-28 19:44:00 +0000 UTC[csr] Generating kubelet configuration[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"Node join complete:* Certificate signing request sent to master and response  received.* Kubelet informed of new secure connection details.Run 'kubectl get nodes' on the master to see this machine join.

安装完成后可以查看下状态，未安装网络组件，所以全部都是NotReady状态：

NAME       STATUS        AGE      VERSIONmaster01   NotReady      1d       v1.7.6master02   NotReady      1d       v1.7.6master03   NotReady      1d       v1.7.6node01     NotReady      1d       v1.7.6node02     NotReady      1d       v1.7.6node03     NotReady      1d       v1.7.6

8 安装Calico网络

网络组件选择很多，可以根据自己的需要选择calico、weave、flannel，calico性能最好，flannel的vxlan也不错，默认的UDP性能较差，weave的性能比较差，测试环境用下可以，生产环境不建议使用。Addons中有配置好的yaml，所以本文中尝试calico网络，。

kubectl apply -f https://docs.projectcalico.org/v2.5/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml

如果使用了外部etcd，去掉etcd相关配置内容，并修改etcd_endpoints: [ETCD_ENDPOINTS]：

# Calico Version v2.5.1# https://docs.projectcalico.org/v2.5/releases#v2.5.1# This manifest includes the following component versions:#   calico/node:v2.5.1#   calico/cni:v1.10.0#   calico/kube-policy-controller:v0.7.0# This ConfigMap is used to configure a self-hosted Calico installation.kind: ConfigMapapiVersion: v1metadata:  name: calico-config  namespace: kube-systemdata:  # The location of your etcd cluster.  This uses the Service clusterIP defined below.  etcd_endpoints: "http://172.16.2.1:2379,http://172.16.2.2:2379,http://172.16.2.3:2379"  # Configure the Calico backend to use.  calico_backend: "bird"  # The CNI network configuration to install on each node.  cni_network_config: |-    {        "name": "k8s-pod-network",        "cniVersion": "0.1.0",        "type": "calico",        "etcd_endpoints": "__ETCD_ENDPOINTS__",        "log_level": "info",        "mtu": 1500,        "ipam": {            "type": "calico-ipam"        },        "policy": {            "type": "k8s",             "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",             "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"        },        "kubernetes": {            "kubeconfig": "/etc/cni/net.d/__KUBECONFIG_FILENAME__"        }    }---# This manifest installs the calico/node container, as well# as the Calico CNI plugins and network config on# each master and worker node in a Kubernetes cluster.kind: DaemonSetapiVersion: extensions/v1beta1metadata:  name: calico-node  namespace: kube-system  labels:    k8s-app: calico-nodespec:  selector:    matchLabels:      k8s-app: calico-node  template:    metadata:      labels:        k8s-app: calico-node      annotations:        # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler        # reserves resources for critical add-on pods so that they can be rescheduled after        # a failure.  This annotation works in tandem with the toleration below.        scheduler.alpha.kubernetes.io/critical-pod: ''    spec:      hostNetwork: true      tolerations:      - key: node-role.kubernetes.io/master        effect: NoSchedule      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.      # This, along with the annotation above marks this pod as a critical add-on.      - key: CriticalAddonsOnly        operator: Exists      serviceAccountName: calico-cni-plugin      containers:        # Runs calico/node container on each Kubernetes node.  This        # container programs network policy and routes on each        # host.        - name: calico-node          image: quay.io/calico/node:v2.5.1          env:            # The location of the Calico etcd cluster.            - name: ETCD_ENDPOINTS              valueFrom:                configMapKeyRef:                  name: calico-config                  key: etcd_endpoints            # Enable BGP.  Disable to enforce policy only.            - name: CALICO_NETWORKING_BACKEND              valueFrom:                configMapKeyRef:                  name: calico-config                  key: calico_backend            # Cluster type to identify the deployment type            - name: CLUSTER_TYPE              value: "kubeadm,bgp"            # Disable file logging so `kubectl logs` works.            - name: CALICO_DISABLE_FILE_LOGGING              value: "true"            # Set Felix endpoint to host default action to ACCEPT.            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION              value: "ACCEPT"            # Configure the IP Pool from which Pod IPs will be chosen.            - name: CALICO_IPV4POOL_CIDR              value: "10.68.0.0/16"            - name: CALICO_IPV4POOL_IPIP              value: "always"            # Disable IPv6 on Kubernetes.            - name: FELIX_IPV6SUPPORT              value: "false"            # Set MTU for tunnel device used if ipip is enabled            - name: FELIX_IPINIPMTU              value: "1440"            # Set Felix logging to "info"            - name: FELIX_LOGSEVERITYSCREEN              value: "info"            # Auto-detect the BGP IP address.            - name: IP              value: ""            - name: FELIX_HEALTHENABLED              value: "true"          securityContext:            privileged: true          resources:            requests:              cpu: 250m          livenessProbe:            httpGet:              path: /liveness              port: 9099            periodSeconds: 10            initialDelaySeconds: 10            failureThreshold: 6          readinessProbe:            httpGet:              path: /readiness              port: 9099            periodSeconds: 10          volumeMounts:            - mountPath: /lib/modules              name: lib-modules              readOnly: true            - mountPath: /var/run/calico              name: var-run-calico              readOnly: false        # This container installs the Calico CNI binaries        # and CNI network config file on each node.        - name: install-cni          image: quay.io/calico/cni:v1.10.0          command: ["/install-cni.sh"]          env:            # The location of the Calico etcd cluster.            - name: ETCD_ENDPOINTS              valueFrom:                configMapKeyRef:                  name: calico-config                  key: etcd_endpoints            # The CNI network config to install on each node.            - name: CNI_NETWORK_CONFIG              valueFrom:                configMapKeyRef:                  name: calico-config                  key: cni_network_config          volumeMounts:            - mountPath: /host/opt/cni/bin              name: cni-bin-dir            - mountPath: /host/etc/cni/net.d              name: cni-net-dir      volumes:        # Used by calico/node.        - name: lib-modules          hostPath:            path: /lib/modules        - name: var-run-calico          hostPath:            path: /var/run/calico        # Used to install CNI.        - name: cni-bin-dir          hostPath:            path: /opt/cni/bin        - name: cni-net-dir          hostPath:            path: /etc/cni/net.d---# This manifest deploys the Calico policy controller on Kubernetes.# See https://github.com/projectcalico/k8s-policyapiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: calico-policy-controller  namespace: kube-system  labels:    k8s-app: calico-policyspec:  # The policy controller can only have a single active instance.  replicas: 1  strategy:    type: Recreate  template:    metadata:      name: calico-policy-controller      namespace: kube-system      labels:        k8s-app: calico-policy-controller      annotations:        # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler        # reserves resources for critical add-on pods so that they can be rescheduled after        # a failure.  This annotation works in tandem with the toleration below.        scheduler.alpha.kubernetes.io/critical-pod: ''    spec:      # The policy controller must run in the host network namespace so that      # it isn't governed by policy that would prevent it from working.      hostNetwork: true      tolerations:      - key: node-role.kubernetes.io/master        effect: NoSchedule      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.      # This, along with the annotation above marks this pod as a critical add-on.      - key: CriticalAddonsOnly        operator: Exists      serviceAccountName: calico-policy-controller      containers:        - name: calico-policy-controller          image: quay.io/calico/kube-policy-controller:v0.7.0          env:            # The location of the Calico etcd cluster.            - name: ETCD_ENDPOINTS              valueFrom:                configMapKeyRef:                  name: calico-config                  key: etcd_endpoints            # The location of the Kubernetes API.  Use the default Kubernetes            # service for API access.            - name: K8S_API              value: "https://kubernetes.default:443"            # Since we're running in the host namespace and might not have KubeDNS            # access, configure the container's /etc/hosts to resolve            # kubernetes.default to the correct service clusterIP.            - name: CONFIGURE_ETC_HOSTS              value: "true"---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata:  name: calico-cni-pluginroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: calico-cni-pluginsubjects:- kind: ServiceAccount  name: calico-cni-plugin  namespace: kube-system---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1beta1metadata:  name: calico-cni-plugin  namespace: kube-systemrules:  - apiGroups: [""]    resources:      - pods      - nodes    verbs:      - get---apiVersion: v1kind: ServiceAccountmetadata:  name: calico-cni-plugin  namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata:  name: calico-policy-controllerroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: calico-policy-controllersubjects:- kind: ServiceAccount  name: calico-policy-controller  namespace: kube-system---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1beta1metadata:  name: calico-policy-controller  namespace: kube-systemrules:  - apiGroups:    - ""    - extensions    resources:      - pods      - namespaces      - networkpolicies    verbs:      - watch      - list---apiVersion: v1kind: ServiceAccountmetadata:  name: calico-policy-controller  namespace: kube-system

检查各节点组件运行状态：

NAME                                        READY     STATUS     RESTARTS   AGEcalico-node-0cjx5                           2/2       Running    0          1dcalico-node-1vj9s                           2/2       Running    0          1dcalico-node-222v0                           2/2       Running    0          1dcalico-node-7nqj7                           2/2       Running    0          1dcalico-node-7tvh9                           2/2       Running    0          1dcalico-node-86313                           2/2       Running    2          1dcalico-policy-controller-3691403067-43wm6   1/1       Running    3          1dkube-apiserver-master01                     1/1       Running    3          1dkube-apiserver-master02                     1/1       Running    1          1dkube-apiserver-master03                     1/1       Running    1          1dkube-controller-manager-master01            1/1       Running    4          1dkube-controller-manager-master02            1/1       Running    2          1dkube-controller-manager-master03            1/1       Running    2          1dkube-dns-4099109879-3hqtq                   3/3       Running    0          1dkube-proxy-43j51                            1/1       Running    1          1dkube-proxy-4z8mx                            1/1       Running    0          1dkube-proxy-8w1xh                            1/1       Running    0          1dkube-proxy-g2hv8                            1/1       Running    1          1dkube-proxy-hzkmc                            1/1       Running    1          1dkube-proxy-l91xr                            1/1       Running    0          1dkube-scheduler-master01                     1/1       Running    4          1dkube-scheduler-master02                     1/1       Running    2          1dkube-scheduler-master03                     1/1       Running    2          1d

说明：kube-dns需要等calico配置完成后才是running状态。

8 DNS集群部署

删除原单点kube-dns

kubectl delete deploy kube-dns -n kube-system

部署多实例的kube-dns集群，参考配置kube-dns.yml：

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  labels:    k8s-app: kube-dns  name: kube-dns  namespace: kube-systemspec:  replicas: 3  selector:    matchLabels:      k8s-app: kube-dns  strategy:    rollingUpdate:      maxSurge: 10%      maxUnavailable: 0    type: RollingUpdate  template:    metadata:      annotations:        scheduler.alpha.kubernetes.io/critical-pod: ""      creationTimestamp: null      labels:        k8s-app: kube-dns    spec:      affinity:        nodeAffinity:          requiredDuringSchedulingIgnoredDuringExecution:            nodeSelectorTerms:            - matchExpressions:              - key: beta.kubernetes.io/arch                operator: In                values:                - amd64        podAntiAffinity:          requiredDuringSchedulingIgnoredDuringExecution:            - labelSelector:                matchExpressions:                  - key: k8s-app                    operator: In                    values:                    - kube-dns              topologyKey: "kubernetes.io/hostname"      containers:      - args:        - --domain=cluster.local.        - --dns-port=10053        - --config-dir=/kube-dns-config        - --v=2        env:        - name: PROMETHEUS_PORT          value: "10055"        image: cloudnil/k8s-dns-kube-dns-amd64:1.14.4        imagePullPolicy: IfNotPresent        livenessProbe:          failureThreshold: 5          httpGet:            path: /healthcheck/kubedns            port: 10054            scheme: HTTP          initialDelaySeconds: 60          periodSeconds: 10          successThreshold: 1          timeoutSeconds: 5        name: kubedns        ports:        - containerPort: 10053          name: dns-local          protocol: UDP        - containerPort: 10053          name: dns-tcp-local          protocol: TCP        - containerPort: 10055          name: metrics          protocol: TCP        readinessProbe:          failureThreshold: 3          httpGet:            path: /readiness            port: 8081            scheme: HTTP          initialDelaySeconds: 3          periodSeconds: 10          successThreshold: 1          timeoutSeconds: 5        resources:          limits:            memory: 170Mi          requests:            cpu: 100m            memory: 70Mi        terminationMessagePath: /dev/termination-log        terminationMessagePolicy: File        volumeMounts:        - mountPath: /kube-dns-config          name: kube-dns-config      - args:        - -v=2        - -logtostderr        - -configDir=/etc/k8s/dns/dnsmasq-nanny        - -restartDnsmasq=true        - --        - -k        - --cache-size=1000        - --log-facility=-        - --server=/cluster.local/127.0.0.1#10053        - --server=/in-addr.arpa/127.0.0.1#10053        - --server=/ip6.arpa/127.0.0.1#10053        image: cloudnil/k8s-dns-dnsmasq-nanny-amd64:1.14.4        imagePullPolicy: IfNotPresent        livenessProbe:          failureThreshold: 5          httpGet:            path: /healthcheck/dnsmasq            port: 10054            scheme: HTTP          initialDelaySeconds: 60          periodSeconds: 10          successThreshold: 1          timeoutSeconds: 5        name: dnsmasq        ports:        - containerPort: 53          name: dns          protocol: UDP        - containerPort: 53          name: dns-tcp          protocol: TCP        resources:          requests:            cpu: 150m            memory: 20Mi        terminationMessagePath: /dev/termination-log        terminationMessagePolicy: File        volumeMounts:        - mountPath: /etc/k8s/dns/dnsmasq-nanny          name: kube-dns-config      - args:        - --v=2        - --logtostderr        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A        image: cloudnil/k8s-dns-sidecar-amd64:1.14.4        imagePullPolicy: IfNotPresent        livenessProbe:          failureThreshold: 5          httpGet:            path: /metrics            port: 10054            scheme: HTTP          initialDelaySeconds: 60          periodSeconds: 10          successThreshold: 1          timeoutSeconds: 5        name: sidecar        ports:        - containerPort: 10054          name: metrics          protocol: TCP        resources:          requests:            cpu: 10m            memory: 20Mi        terminationMessagePath: /dev/termination-log        terminationMessagePolicy: File      dnsPolicy: Default      restartPolicy: Always      schedulerName: default-scheduler      securityContext: {}      serviceAccount: kube-dns      serviceAccountName: kube-dns      terminationGracePeriodSeconds: 30      tolerations:      - effect: NoSchedule        key: node-role.kubernetes.io/master      - key: CriticalAddonsOnly        operator: Exists      volumes:      - configMap:          defaultMode: 420          name: kube-dns          optional: true        name: kube-dns-config

9 部署Dashboard

下载kubernetes-dashboard.yaml

curl -O https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml

修改配置内容，部署到default的namespace，增加ingress配置，后边配置了nginx-ingress后就可以直接绑定域名访问了。

apiVersion: v1kind: ServiceAccountmetadata:  labels:    k8s-app: kubernetes-dashboard  name: kubernetes-dashboard  namespace: default---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata:  name: kubernetes-dashboard  labels:    k8s-app: kubernetes-dashboardroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: cluster-adminsubjects:- kind: ServiceAccount  name: kubernetes-dashboard  namespace: default---kind: DeploymentapiVersion: extensions/v1beta1metadata:  labels:    k8s-app: kubernetes-dashboard  name: kubernetes-dashboard  namespace: defaultspec:  replicas: 1  revisionHistoryLimit: 10  selector:    matchLabels:      k8s-app: kubernetes-dashboard  template:    metadata:      labels:        k8s-app: kubernetes-dashboard    spec:      containers:      - name: kubernetes-dashboard        image: cloudnil/kubernetes-dashboard-amd64:v1.7.0        ports:        - containerPort: 9090          protocol: TCP        args:        livenessProbe:          httpGet:            path: /            port: 9090          initialDelaySeconds: 30          timeoutSeconds: 30      serviceAccountName: kubernetes-dashboard      # Comment the following tolerations if Dashboard must not be deployed on master      tolerations:      - key: node-role.kubernetes.io/master        effect: NoSchedule---kind: ServiceapiVersion: v1metadata:  labels:    k8s-app: kubernetes-dashboard  name: kubernetes-dashboard  namespace: defaultspec:  ports:  - port: 80    targetPort: 9090  selector:    k8s-app: kubernetes-dashboard---apiVersion: extensions/v1beta1kind: Ingressmetadata:  name: dashboard-ingress  namespace: defaultspec:  rules:  - host: dashboard.cloudnil.com    http:      paths:      - path: /        backend:          serviceName: kubernetes-dashboard          servicePort: 80

10 Dashboard服务暴露到公网

kubernetes中的Service暴露到外部有三种方式，分别是：

• LoadBlancer Service
• NodePort Service
• Ingress

LoadBlancer Service是kubernetes深度结合云平台的一个组件；当使用LoadBlancer Service暴露服务时，实际上是通过向底层云平台申请创建一个负载均衡器来向外暴露服务；目前LoadBlancer Service支持的云平台已经相对完善，比如国外的GCE、DigitalOcean，国内的 阿里云，私有云 Openstack 等等，由于LoadBlancer Service深度结合了云平台，所以只能在一些云平台上来使用。

NodePort Service顾名思义，实质上就是通过在集群的每个node上暴露一个端口，然后将这个端口映射到某个具体的service来实现的，虽然每个node的端口有很多(0~65535)，但是由于安全性和易用性(服务多了就乱了，还有端口冲突问题)实际使用可能并不多。

Ingress可以实现使用nginx等开源的反向代理负载均衡器实现对外暴露服务，可以理解Ingress就是用于配置域名转发的一个东西，在nginx中就类似upstream，它与ingress-controller结合使用，通过ingress-controller监控到pod及service的变化，动态地将ingress中的转发信息写到诸如nginx、apache、haproxy等组件中实现方向代理和负载均衡。

10.1 部署Nginx-ingress-controller

Nginx-ingress-controller是kubernetes官方提供的集成了Ingress-controller和Nginx的一个docker镜像。

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: default-http-backend  labels:    k8s-app: default-http-backend  namespace: defaultspec:  replicas: 1  template:    metadata:      labels:        k8s-app: default-http-backend    spec:      terminationGracePeriodSeconds: 60      containers:      - name: default-http-backend        image: cloudnil/defaultbackend:1.0        livenessProbe:          httpGet:            path: /healthz            port: 8080            scheme: HTTP          initialDelaySeconds: 30          timeoutSeconds: 5        ports:        - containerPort: 8080        resources:          limits:            cpu: 10m            memory: 20Mi          requests:            cpu: 10m            memory: 20Mi---apiVersion: v1kind: Servicemetadata:  name: default-http-backend  labels:    k8s-app: default-http-backend  namespace: defaultspec:  ports:  - port: 80    targetPort: 8080  selector:    k8s-app: default-http-backend---apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: nginx-ingress-controller  labels:    k8s-app: nginx-ingress-controller  namespace: defaultspec:  replicas: 1  template:    metadata:      labels:        k8s-app: nginx-ingress-controller    spec:      hostNetwork: true      nodeName: master      terminationGracePeriodSeconds: 60      serviceAccountName: nginx-ingress-controller      containers:      - image: cloudnil/nginx-ingress-controller:0.9.0-beta.13        name: nginx-ingress-controller        readinessProbe:          httpGet:            path: /healthz            port: 10254            scheme: HTTP        livenessProbe:          httpGet:            path: /healthz            port: 10254            scheme: HTTP          initialDelaySeconds: 10          timeoutSeconds: 1        ports:        - containerPort: 80          hostPort: 80        - containerPort: 443          hostPort: 443        env:          - name: POD_NAME            valueFrom:              fieldRef:                fieldPath: metadata.name          - name: POD_NAMESPACE            valueFrom:              fieldRef:                fieldPath: metadata.namespace        args:        - /nginx-ingress-controller        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend---apiVersion: v1kind: ServiceAccountmetadata:  name: nginx-ingress-controller  namespace: default---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata:  name: view-services-clusterroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: cluster-adminsubjects:- kind: ServiceAccount  name: nginx-ingress-controller  namespace: default

部署完Nginx-ingress-controller后，解析域名dashboard.cloudnil.com到node02的外网IP，就可以使用dashboard.cloudnil.com访问dashboard。

11 注意事项

kubeadm目前还在开发测试阶段，不建议在生产环境中使用kubeadm部署kubernetes环境。此外，使用kubeadm是需要注意以下几点：

11.1 暴露主机端口

POD实例配置中的HostPort和HostIP参数无法用于使用了CNI网络插件的kubernetes集群环境，如果需要暴露容器到主机端口，可以使用NodePort或者HostNetwork。

11.2 CentOS环境路由错误

RHEL/CentOS7 环境中iptables的策略关系，会导致路由通讯错误，需要手动调整iptables的桥接设置：

# cat /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1

11.3 Token丢失

Master节点部署完成之后，会输出一个token用于minion节点的配置链接，不过这个token没有很方便的查看方式，导致此日志输出关闭后，没有token无法join minion节点，可以通过下述方式查看token：

kubectl -n kube-system get secret clusterinfo -o yaml | grep token-map | awk '{print $2}' | base64 --decode | sed "s|{||g;s|}||g;s|:|.|g;s/\"//g;" | xargs echo

建议提前使用kubeadm token命令生成token，然后在执行kubeadm init和kubeadm join的使用通过--token指定token。

11.4 Vagrant中主机名的问题

如果使用Vagrant虚拟化环境部署kubernetes，首先得确保hostname -i能够获取正确的通讯IP，默认情况下，如果/etc/hosts中未配置主机名与IP的对应关系，kubelet会取第一个非lo网卡作为通讯入口，若这个网卡不做了NAT桥接的网卡，那安装就会出现问题。

11.5 Master节点上kubeconfig未加载的问题

kubectl默认应该是会加载配置文件：/etc/kubernetes/admin.conf，但是本次部署后，kubectl未加载该配置文件，可以添加一条环境变量：export KUBECONFIG=/etc/kubernetes/admin.conf，问题解决。