使用kuberspay部署高可用kubernetes集群

0 环境

环境：

主机名 IP k8s-node01 172.16.120.151 k8s-node02 172.16.120.152 k8s-node03 172.16.120.153 ansible-client

==mac os x固定vware虚拟机IP
sudo vi /Library/Preferences/VMware\ Fusion/vmnet8/dhcpd.conf
在文件末尾添加==

host CentOS01{    hardware ethernet 00:0C:29:15:5C:F1;    fixed-address 172.16.120.151;}host CentOS02{    hardware ethernet 00:0C:29:D1:C4:9A;    fixed-address 172.16.120.152;}host CentOS03{    hardware ethernet 00:0C:29:C2:A6:93;    fixed-address 172.16.120.153;}

• centos01为固定ip虚拟机的名称
• hardware ethernet 硬件地址
• fixed-address 固定ip地址

ip地址取值范围必须在hdcpd.conf给定的范围内,配置完成后重启vware。

设置主机名：

hostnamectl --static set-hostname  k8s-node01hostnamectl --static set-hostname  k8s-node02hostnamectl --static set-hostname  k8s-node03

关闭防火墙：

systemctl disable firewalldsystemctl stop firewalldsed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

由于kubespay安装方式会检测docker是否安装，如果没有安装会安装docker，但是使用的源是https://yum.dockerproject.org/repo/main/centos/7，速度会比较慢，建议提前安装好。

使用阿里云yum镜像,docker安装速度快

#docker yum源cat >> /etc/yum.repos.d/docker.repo <<EOF[docker-repo]name=Docker Repositorybaseurl=http://mirrors.aliyun.com/docker-engine/yum/repo/main/centos/7enabled=1gpgcheck=0EOF

同时配置好阿里云加速器

mkdir -p /etc/dockertee /etc/docker/daemon.json <<-'EOF'{  "registry-mirrors": ["https://5md0553g.mirror.aliyuncs.com"]}EOF

手动安装docker:

#查看docker版本yum list docker-engine –showduplicates#安装dockeryum install -y docker-engine-1.13.1-1.el7.centos.x86_64

1 安装 ansible

在ansible的控制服务器ansible-client上安装ansible。

# 安装 python 及 epelyum install -y epel-release python-pip python34 python34-pip# 安装 ansible(必须先安装 epel 源再安装 ansible)yum install -y ansible

托管节点python的版本需要大于2.5。

ansible不支持windows，其他系统的安装方式可以查阅ansible官方网站。

Ansible中文权威指南：
http://ansible-tran.readthedocs.io/en/latest/

2 设置免密登录

在ansible-client执行 ssh-keygen -t rsa 生成密钥对

ssh-keygen -t rsa -P ''

将~/.ssh/id_rsa.pub复制到其他所有节点，这样ansible-client到其他所有节点可以免密登录

IP=(172.16.120.151 172.16.120.152 172.16.120.153)for x in ${IP[*]}; do ssh-copy-id -i ~/.ssh/id_rsa.pub $x; done

要使用root权限执行上面操作。

3 下载kuberspay源码

可以从主分支下载：

git clone https://github.com/kubernetes-incubator/kubespray.git

也可以下载发布版本：

wget https://github.com/kubernetes-incubator/kubespray/archive/v2.1.2.tar.gz

本文采用v2.1.2 发布版本安装。

4 将grc.io和quay.io的镜像上传到阿里云

kuberspay涉及到的镜像。

quay.io/coreos/hyperkube:v1.7.3_coreos.0quay.io/coreos/etcd:v3.2.4quay.io/calico/ctl:v1.4.0quay.io/calico/node:v2.4.1quay.io/calico/cni:v1.10.0quay.io/kube-policy-controller:v0.7.0quay.io/calico/routereflector:v0.3.0quay.io/coreos/flannel:v0.8.0quay.io/coreos/flannel-cni:v0.2.0quay.io/l23network/k8s-netchecker-agent:v1.0quay.io/l23network/k8s-netchecker-server:v1.0weaveworks/weave-kube:2.0.1weaveworks/weave-npc:2.0.1gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1gcr.io/google_containers/fluentd-elasticsearch:1.22gcr.io/google_containers/kibana:v4.6.1gcr.io/google_containers/elasticsearch:v2.4.1gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.2gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.2gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2gcr.io/google_containers/pause-amd64:3.0gcr.io/kubernetes-helm/tiller:v2.2.2gcr.io/google_containers/heapster-grafana-amd64:v4.4.1gcr.io/google_containers/heapster-amd64:v1.4.0gcr.io/google_containers/heapster-influxdb-amd64:v1.1.1gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.11gcr.io/google_containers/defaultbackend:1.3

通过grc.io的到阿里云：

#!/bin/bashset -o errexitset -o nounsetset -o pipefailkubednsautoscaler_version=1.1.1dns_version=1.14.2kube_pause_version=3.0dashboard_version=v1.6.3fluentd_es_version=1.22kibana_version=v4.6.1elasticsearch_version=v2.4.1heapster_version=v1.4.0heapster_grafana_version=v4.4.1heapster_influxdb_version=v1.1.1nginx_ingress_version=0.9.0-beta.11defaultbackend_version=1.3GCR_URL=gcr.io/google_containersALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/szss_k8simages=(cluster-proportional-autoscaler-amd64:${kubednsautoscaler_version}k8s-dns-sidecar-amd64:${dns_version}k8s-dns-kube-dns-amd64:${dns_version}k8s-dns-dnsmasq-nanny-amd64:${dns_version}pause-amd64:${kube_pause_version}kubernetes-dashboard-amd64:${dashboard_version}fluentd-elasticsearch:${fluentd_es_version}kibana:${kibana_version}elasticsearch:${elasticsearch_version}fluentd-elasticsearch:${fluentd_es_version}kibana:${kibana_version}heapster-amd64:${heapster_version}heapster-grafana-amd64:${heapster_grafana_version}heapster-influxdb-amd64:${heapster_influxdb_version}nginx-ingress-controller:${nginx_ingress_version}defaultbackend:${defaultbackend_version})for imageName in ${images[@]} ; do  docker pull $GCR_URL/$imageName  docker tag $GCR_URL/$imageName $ALIYUN_URL/$imageName  docker push $ALIYUN_URL/$imageName  docker rmi $ALIYUN_URL/$imageNamedone

通过quay.io的镜像到阿里云：

QUAY_URL=quay.ioALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/szss_quay_io# master分支#images=(#coreos/hyperkube:v1.7.3_coreos.0#coreos/etcd:v3.2.4#coreos/flannel:v0.8.0#coreos/flannel-cni:v0.2.0#calico/kube-policy-controller:v0.7.0#calico/ctl:v1.4.0#calico/node:v2.4.1#calico/cni:v1.10.0#calico/routereflector:v0.3.0#l23network/k8s-netchecker-agent:v1.0#l23network/k8s-netchecker-server:v1.0#)# kuberspay v2.1.2版本images=(coreos/hyperkube:v1.6.7_coreos.0coreos/etcd:v3.2.4coreos/flannel:v0.8.0coreos/flannel-cni:v0.2.0calico/kube-policy-controller:v0.5.4calico/ctl:v1.1.3calico/node:v1.1.3calico/cni:v1.8.0calico/routereflector:v0.3.0l23network/k8s-netchecker-agent:v1.0l23network/k8s-netchecker-server:v1.0)for imageName in ${images[@]} ; do  docker pull $QUAY_URL/$imageName  docker tag $QUAY_URL/$imageName $ALIYUN_URL/${imageName/\//-}  docker push $ALIYUN_URL/${imageName/\//-}  docker rmi $ALIYUN_URL/${imageName/\//-}done

5 镜像替换

在kuberspay源码源代码中搜索包含 gcr.io/google_containers 和 quay.io 镜像的文件，并替换为我们之前已经上传到阿里云的进行，替换脚步如下：

grc_image_files=(./kubespray/extra_playbooks/roles/dnsmasq/templates/dnsmasq-autoscaler.yml./kubespray/extra_playbooks/roles/download/defaults/main.yml./kubespray/extra_playbooks/roles/kubernetes-apps/ansible/defaults/main.yml./kubespray/roles/download/defaults/main.yml./kubespray/roles/dnsmasq/templates/dnsmasq-autoscaler.yml./kubespray/roles/kubernetes-apps/ansible/defaults/main.yml)for file in ${grc_image_files[@]} ; do    sed -i 's/gcr.io\/google_containers/registry.cn-hangzhou.aliyuncs.com\/szss_k8s/g' $filedonequay_image_files=(./kubespray/extra_playbooks/roles/download/defaults/main.yml./kubespray/roles/download/defaults/main.yml)for file in ${quay_image_files[@]} ; do    sed -i 's/quay.io\/coreos\//registry.cn-hangzhou.aliyuncs.com\/szss_quay_io\/coreos-/g' $file    sed -i 's/quay.io\/calico\//registry.cn-hangzhou.aliyuncs.com\/szss_quay_io\/calico-/g' $file    sed -i 's/quay.io\/l23network\//registry.cn-hangzhou.aliyuncs.com\/szss_quay_io\/l23network-/g' $filedone

如果在mac os x执行脚本，需要在sed -i 后面添加一个空字符串，例如sed -i ” ‘s/a/b/g’ file

6 配置文件内容

可以对basic_auth的密码进行修改，网络插件默认calico，可替换成weave或flannel，还可以配置是否安装helm和efk。

下面的的配置为kuberspay 2.1.2的配置。

$vi ~/kubespray/inventory/group_vars/k8s-cluster.yml# Kubernetes configuration dirs and system namespace.# Those are where all the additional config stuff goes# the kubernetes normally puts in /srv/kubernets.# This puts them in a sane location and namespace.# Editting those values will almost surely break something.kube_config_dir: /etc/kuberneteskube_script_dir: "{{ bin_dir }}/kubernetes-scripts"kube_manifest_dir: "{{ kube_config_dir }}/manifests"system_namespace: kube-system# Logging directory (sysvinit systems)kube_log_dir: "/var/log/kubernetes"# This is where all the cert scripts and certs will be locatedkube_cert_dir: "{{ kube_config_dir }}/ssl"# This is where all of the bearer tokens will be storedkube_token_dir: "{{ kube_config_dir }}/tokens"# This is where to save basic auth filekube_users_dir: "{{ kube_config_dir }}/users"kube_api_anonymous_auth: false## Change this to use another Kubernetes version, e.g. a current beta releasekube_version: v1.6.7# Where the binaries will be downloaded.# Note: ensure that you've enough disk space (about 1G)local_release_dir: "/tmp/releases"# Random shifts for retrying failed ops like pushing/downloadingretry_stagger: 5# This is the group that the cert creation scripts chgrp the# cert files to. Not really changable...kube_cert_group: kube-cert# Cluster Loglevel configurationkube_log_level: 2# Users to create for basic auth in Kubernetes API via HTTP# Optionally add groups for userkube_api_pwd: "changeme"kube_users:  kube:    pass: "{{kube_api_pwd}}"    role: admin  root:    pass: "{{kube_api_pwd}}"    role: admin    # groups:    #   - system:masters## It is possible to activate / deactivate selected authentication methods (basic auth, static token auth)#kube_oidc_auth: false#kube_basic_auth: false#kube_token_auth: false## Variables for OpenID Connect Configuration https://kubernetes.io/docs/admin/authentication/## To use OpenID you have to deploy additional an OpenID Provider (e.g Dex, Keycloak, ...)# kube_oidc_url: https:// ...# kube_oidc_client_id: kubernetes## Optional settings for OIDC# kube_oidc_ca_file: {{ kube_cert_dir }}/ca.pem# kube_oidc_username_claim: sub# kube_oidc_groups_claim: groups# Choose network plugin (calico, weave or flannel)# Can also be set to 'cloud', which lets the cloud provider setup appropriate routingkube_network_plugin: calico# weave's network password for encryption# if null then no network encryption# you can use --extra-vars to pass the password in command lineweave_password: EnterPasswordHere# Weave uses consensus mode by default# Enabling seed mode allow to dynamically add or remove hosts# https://www.weave.works/docs/net/latest/ipam/weave_mode_seed: false# This two variable are automatically changed by the weave's role, do not manually change these values# To reset values :# weave_seed: uninitialized# weave_peers: uninitializedweave_seed: uninitializedweave_peers: uninitialized# Enable kubernetes network policiesenable_network_policy: false# Kubernetes internal network for services, unused block of space.kube_service_addresses: 10.233.0.0/18# internal network. When used, it will assign IP# addresses from this range to individual pods.# This network must be unused in your network infrastructure!kube_pods_subnet: 10.233.64.0/18# internal network node size allocation (optional). This is the size allocated# to each node on your network.  With these defaults you should have# room for 4096 nodes with 254 pods per node.kube_network_node_prefix: 24# The port the API Server will be listening on.kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"kube_apiserver_port: 6443 # (https)kube_apiserver_insecure_port: 8080 # (http)# DNS configuration.# Kubernetes cluster name, also will be used as DNS domaincluster_name: cluster.local# Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet podsndots: 2# Can be dnsmasq_kubedns, kubedns or nonedns_mode: kubedns# Can be docker_dns, host_resolvconf or noneresolvconf_mode: docker_dns# Deploy netchecker app to verify DNS resolve as an HTTP servicedeploy_netchecker: false# Ip address of the kubernetes skydns serviceskydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"dns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(2)|ipaddr('address') }}"dns_domain: "{{ cluster_name }}"# Path used to store Docker datadocker_daemon_graph: "/var/lib/docker"## A string of extra options to pass to the docker daemon.## This string should be exactly as you wish it to appear.## An obvious use case is allowing insecure-registry access## to self hosted registries like so:docker_options: "--insecure-registry={{ kube_service_addresses }} --graph={{ docker_daemon_graph }}  {{ docker_log_opts }}"docker_bin_dir: "/usr/bin"# Settings for containerized control plane (etcd/kubelet/secrets)etcd_deployment_type: dockerkubelet_deployment_type: dockercert_management: scriptvault_deployment_type: docker# K8s image pull policy (imagePullPolicy)k8s_image_pull_policy: IfNotPresent# Monitoring apps for k8sefk_enabled: false# Helm deploymenthelm_enabled: false# dnsmasq# dnsmasq_upstream_dns_servers:#  - /resolvethiszone.with/10.0.4.250#  - 8.8.8.8#  Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (default true)# kubelet_cgroups_per_qos: true# A comma separated list of levels of node allocatable enforcement to be enforced by kubelet.# Acceptible options are 'pods', 'system-reserved', 'kube-reserved' and ''. Default is "".# kubelet_enforce_node_allocatable: pods

7 生成集群配置

yum install -y python-pip python34 python34-pip

# 定义集群IPIP=(172.16.120.151172.16.120.152172.16.120.153)# 利用kubespray自带的python脚本生成配置CONFIG_FILE=./kubespray/inventory/inventory.cfg python3 ./kubespray/contrib/inventory_builder/inventory.py ${IP[*]}

集群配置如下：

$cat ./kubespray/inventory/inventory.cfg [all]node1    ansible_host=172.16.120.151 ip=172.16.120.151node2    ansible_host=172.16.120.152 ip=172.16.120.152node3    ansible_host=172.16.120.153 ip=172.16.120.153[kube-master]node1    node2    [kube-node]node1    node2    node3    [etcd]node1    node2    node3    [k8s-cluster:children]kube-node    kube-master      [calico-rr][vault]node1    node2    node3

8 安装集群

cd kubesprayansible-playbook -i inventory/inventory.cfg cluster.yml -b -v --private-key=~/.ssh/id_rsa

9 troubles shooting

错误1：在安装过程中报错：

fatal: [node1]: FAILED! => {"failed": true, "msg": "The ipaddr filter requires python-netaddr be installed on the ansible controller"}

需要安装 python-netaddr，安装命令pip install netaddr。

错误2：在安装过程中报错：

{"failed": true, "msg": "The conditional check '{%- set certs = {'sync': False} -%}\n{% if gen_node_certs[inventory_hostname] or\n  (not etcdcert_node.results[0].stat.exists|default(False)) or\n    (not etcdcert_node.results[1].stat.exists|default(False)) or\n      (etcdcert_node.results[1].stat.checksum|default('') != etcdcert_master.files|selectattr(\"path\", \"equalto\", etcdcert_node.results[1].stat.path)|map(attribute=\"checksum\")|first|default('')) -%}\n        {%- set _ = certs.update({'sync': True}) -%}\n{% endif %}\n{{ certs.sync }}' failed. The error was: no test named 'equalto'\n\nThe error appears to have been in '/root/kubespray/roles/etcd/tasks/check_certs.yml': line 57, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"Check_certs | Set 'sync_certs' to true\"\n  ^ here\n"}

升级Jinja2到2.8版本，参考https://github.com/kubernetes-incubator/kubespray/issues/1190， 安装命令pip install –upgrade Jinja2

10 安装失败清理

rm -rf /etc/kubernetes/rm -rf /var/lib/kubeletrm -rf /var/lib/etcdrm -rf /usr/local/bin/kubectlrm -rf /etc/systemd/system/calico-node.servicerm -rf /etc/systemd/system/kubelet.servicesystemctl stop etcd.servicesystemctl disable etcd.servicesystemctl stop calico-node.servicesystemctl disable calico-node.servicedocker stop $(docker ps -q)docker rm $(docker ps -a -q)service docker restart

11 参考

使用kuberspay无坑安装生产级Kubernetes集群: http://www.wisely.top/2017/07/01/no-problem-kubernetes-kuberspay/

使用kuberspay快速部署kubernetes高可用集群： https://mritd.me/2017/03/03/set-up-kubernetes-ha-cluster-by-kargo/

kargo 集群扩展及细粒度配置： https://mritd.me/2017/03/10/kargo-cluster-expansion-and-fine-grained-configuration/

kubespray容器化部署kubernetes高可用集群：
https://kevinguo.me/2017/07/06/kubespray-deploy-kubernetes-1/

使用kargo 安装kubernetes 1.5.3 高可用环境
http://www.jianshu.com/p/ffbfdea089d5