Install Kubernetes From a Scratch

一个完整的kubernetes服务需要许多组件支持. 比如我们需要安装kubelet, api server, scheduler, kube-proxy等等.

不过除了kubelet, docker需要通过system service来启动外, 其他的组件,比如api server, scheduler, etcd等等都可以通过static pod的方式, 让kubernetes本身来为我们启动.

本文着重介绍通过static pod如何启动kubernetes 服务, 并添加一个add-on

节点角色介绍

Master :

Master节点上运行了特定的组件提供集群的管理控制中心,如api-server。
在Master节点上不会运行用户pod/service.

Slave:

区别于Master,除了kubelet, kube-proxy等服务, 还运行用户Pod/Service

准备工作

首先每台虚拟机将要关闭防火墙和SELinux, 同时安装docker服务, 同时生成相关的CA证书, 这里将不在赘述. 具体可参考 [ 部署开源k8s-1.8.5方法]

安装kubelet服务

通过以下kubelete.service文件, 在所有节点启动kubelet服务:

[Unit]Description=Kubernetes KubeletDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=docker.serviceRequires=docker.service[Service]ExecStart=/usr/bin/kubelet \  --address=10.0.10.3 \ ## kublet所服务的ip地址  --hostname-override=10.0.10.3 \  --pod-infra-container-image=172.16.1.41:5000/google_containers/pause:3.0 \  --kubeconfig=/etc/kubernetes/kubeconfig \ ##指定kubeconfig的地址  --require-kubeconfig \  --cert-dir=/etc/kubernetes/ssl \ ##制定CA证书地址  --cluster-dns=10.10.10.10 \  --cluster-domain=cluster.local \  --hairpin-mode=promiscuous-bridge \  --allow-privileged=true \  --logtostderr=true \  --fail-swap-on=false \  --pod-manifest-path=/etc/kubernetes/manifests \ ##制定kubelet监控的pod manifests的地址(重要)  --v=2Restart=on-failureRestartSec=5[Install]WantedBy=multi-user.target

关于kubelet的配置项可参考 [ kubelet官方配置文档]

其中–pod-manifest-path制定kubelet监控的pod manifest的路径

kubelet 会每隔几秒钟(20s)去查看该路径下的pod manifest, 会去启动或更新对应的pod. 常见的Pod manifest格式为yaml或者jason, 官方推荐yaml

执行以下命令,启动kubelet

cp kubelet.service /usr/lib/systemd/system/systemctl daemon-reload && systemctl restart kubelet

配置kubeconfig

kubeConfig定义了kubelet如何访问集群的配置, 其中包含了cluster, user的信息.具体配置如下:

apiVersion: v1kind: Configclusters:  - cluster:      certificate-authority: /etc/kubernetes/ssl/ca.pem      server: https://10.0.10.3:6443    name: kubernetescontexts:  - context:      cluster: kubernetes      user: k8s    name: kubelet-to-kubernetescurrent-context: kubelet-to-kubernetesusers:  - name: k8s    user:      client-certificate: /etc/kubernetes/ssl/admin.pem      client-key: /etc/kubernetes/ssl/admin-key.pemc

上述的kubeConfig定义了一个cluster叫kubernetes, 该集群的server地址, ca证书由clusters域定义.

user定义了用户认证的相关信息. context定义了两者的关系.

在所有节点完成上述配置我们就可以生成对应的Manifest来启动各个组件服务

配置对应组件的Pod Manifest

当docker以及kubelet服务已经启动完毕后,我们只需要生成etcd, kube-controller-manager, kube-proxy, kube-apiserver, kube-scheduler的manifest,并放到刚才kubelet监控的目录, 也就是/etc/kubernetes/manifests.

在Master节点上我们需要配置以下Pod Manifest:

etcd.manifests
kube-apiserver.manifests
kube-controller-manager.manifests
kube-scheduler.manifests

在所有节点我们需要配置:

kube-proxy.manifest

etcd.manifest

apiVersion: v1kind: Podmetadata:  name: k8s-etcd  namespace: kube-systemspec:  hostNetwork: true  containers:  - name: etcd    image: 172.16.1.41:5000/coreos/etcd:3.2.11    imagePullPolicy: IfNotPresent    command:    - etcd    - --data-dir=/var/etcd/data    env:    - name: ETCD_INITIAL_CLUSTER_STATE        value: new    - name: ETCD_NAME      value: ETCD0    - name: ETCD_INITIAL_ADVERTISE_PEER_URLS      value: http://10.0.10.3:2380    - name: ETCD_INITIAL_CLUSTER      value: ETCD0=http://10.0.10.3:2380    - name: ETCD_INITIAL_CLUSTER_TOKEN      value: token    - name: ETCD_LISTEN_PEER_URLS      value: http://0.0.0.0:2380    - name: ETCD_ADVERTISE_CLIENT_URLS      value: http://10.0.10.3:2379    - name: ETCD_LISTEN_CLIENT_URLS      value: http://0.0.0.0:2379    volumeMounts:    - name: varetcd      mountPath: /var/etcd      readOnly: false  volumes:  - name: varetcd    hostPath:      path: /var/etcd/data

关于ETCD的参数配置,可参考 [ ETCD部署]

kube-apiserver.manifest

apiVersion: v1kind: Podmetadata:  name: kube-apiserver  namespace: kube-systemspec:  hostNetwork: true  containers:    - name: apiserver      image: 172.16.1.41:5000/google_containers/kube-apiserver:v1.8.5      imagePullPolicy: IfNotPresent      command:        - kube-apiserver        - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota        - --advertise-address=10.0.10.3        - --allow-privileged=true        - --audit-log-maxage=30        - --audit-log-maxbackup=3        - --audit-log-maxsize=100        - --audit-log-path=/var/log/kubernetes/audit.log        - --authorization-mode=RBAC        - --bind-address=10.0.10.3        - --client-ca-file=/etc/kubernetes/ssl/ca.pem        - --enable-swagger-ui=true        - --etcd-servers=http://10.0.10.3:2379        - --event-ttl=1h        - --kubelet-https=true        - --insecure-bind-address=0.0.0.0        - --runtime-config=rbac.authorization.k8s.io/v1alpha1        - --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem        - --service-cluster-ip-range=10.10.10.1/24        - --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem        - --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem        - --v=2      volumeMounts:      - name: apikubelog        mountPath: /var/log/kubernetes/        readOnly: false      - name: kubernetes        mountPath: /var/run/kubernetes/        readOnly: false      - name: etckube        mountPath: /etc/kubernetes/        readOnly: false  volumes:    - name: apikubelog      hostPath:        path: /var/log/kubernetes/apiserver    - name: kubernetes      hostPath:        path: /var/run/kubernetes/    - name: etckube      hostPath:        path: /etc/kubernetes/

参数解释

– admission-control
定义admission controller(准入)插件, api server会对相应的请求进行校验, 检查请求是否符合插件定义的规则. 详细请参考Adimission Controller

– advertise-address:
广播API Server给所有集群成员的IP地址

–client-ca-file
如果设置后,任何带ca认证的请求都会用指定的ca file进行的比对

–etcd-servers
指定etcd server地址

–kubelet-https
是否使用https建立连接

–service-cluster-ip-range
CIDR标记的IP范围，从中分配IP给服务集群

更多设置解释请参考kube-apiserver配置

kube-controller-manager.manifest

kube-controller-manager是一个daemon, 它会通过api-server不停的轮询监控集群的状态,并在适当的情况下将集群状态迁移到期望的状态
他的manifest如下:

apiVersion: v1kind: Podmetadata:  name: kube-controller-manager  namespace: kube-systemspec:  hostNetwork: true  containers:    - name: kube-controller-manager      image: 172.16.1.41:5000/google_containers/kube-controller-manager:v1.8.5      imagePullPolicy: IfNotPresent      command:        - kube-controller-manager        - --kubeconfig=/etc/kubernetes/kubeconfig ##指定kubeConfig所在地址        - --cluster-name=kubernetes        - --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem ##部署服务所需的秘钥文件        - --root-ca-file=/etc/kubernetes/ssl/ca.pem        - --v=2      volumeMounts:      - name: kubelog        mountPath: /var/log/kubernetes/        readOnly: false      - name: kubernetes        mountPath: /var/run/kubernetes/        readOnly: false      - name: etckube        mountPath: /etc/kubernetes/        readOnly: false  volumes:    - name: kubelog      hostPath:        path: /var/log/kubernetes/apiserver    - name: kubernetes      hostPath:        path: /var/run/kubernetes/    - name: etckube      hostPath:        path: /etc/kubernetes/

kube-scheduler.manifests

kube-scheduler 监视新创建没有分配到Node的Pod，为Pod选择一个Node

apiVersion: v1kind: Podmetadata:  name: kube-scheduler  namespace: kube-systemspec:  hostNetwork: true  containers:    - name: kube-scheduler      image: 172.16.1.41:5000/google_containers/kube-scheduler:v1.8.5      imagePullPolicy: IfNotPresent      command:        - kube-scheduler        - --kubeconfig=/etc/kubernetes/kubeconfig        - --v=2      volumeMounts:      - name: kubelog        mountPath: /var/log/kubernetes/        readOnly: false      - name: kubernetes        mountPath: /var/run/kubernetes/        readOnly: false      - name: etckube        mountPath: /etc/kubernetes/        readOnly: false  volumes:    - name: kubelog      hostPath:        path: /var/log/kubernetes/apiserver    - name: kubernetes      hostPath:        path: /var/run/kubernetes/    - name: etckube      hostPath:        path: /etc/kubernetes/

将以上manifest拷贝到Master节点的kubelet所监控的目录/etc/kubernetes/manifests中

kube-proxy.manifest

在所有节点我们还需要创建kube-proxy.manifest:

kube-proxy通过在主机上维护网络规则并执行连接转发来实现Kubernetes服务抽象

apiVersion: v1kind: Podmetadata:  name: k8s-proxy  namespace: kube-systemspec:  hostNetwork: true  containers:  - name: kube-proxy    image: 172.16.1.41:5000/google_containers/kube-proxy:v1.8.5    imagePullPolicy: Always    command:    - proxy    - --kubeconfig=/etc/kubernetes/kubeconfig    - --v=2    - --masquerade-all ##对所有service ip    - --resource-container=""    securityContext:      privileged: true    volumeMounts:    - name: apikubelog      mountPath: /var/log/kubernetes/      readOnly: false    - name: kubernetes      mountPath: /var/run/kubernetes/      readOnly: false    - name: etckube      mountPath: /etc/kubernetes/      readOnly: false  volumes:    - name: apikubelog      hostPath:        path: /var/log/kubernetes/apiserver    - name: kubernetes      hostPath:        path: /var/run/kubernetes/    - name: etckube      hostPath:        path: /etc/kubernetes/

配置好所有的组件的Manifest, 一个kubernetes集群就启动了, 在本文中有一个master, 1个slave.

如何添加插件Add-on

待添加