k8s集群部署DNS服务发现

环境：

• os: centos
• kubernetes: 1.5.2
• docker: 1.12.5

一、skydns-rc.yaml 文件

apiVersion: v1kind: ReplicationControllermetadata:  name: kube-dns-v9  namespace: default  labels:    k8s-app: kube-dns    version: v9    kubernetes.io/cluster-service: "true"spec:  replicas: 1  selector:    k8s-app: kube-dns    version: v9  template:    metadata:      labels:        k8s-app: kube-dns        version: v9        kubernetes.io/cluster-service: "true"    spec:      containers:      - name: etcd        image: 192.168.100.90:5000/duni/etcd-amd64:3.0.17        imagePullPolicy: IfNotPresent        resources:          limits:            cpu: 100m            memory: 50Mi        command:        - /usr/local/bin/etcd       # - --privileged=true        - -data-dir        - /home/data/etcd        - -listen-client-urls        - http://127.0.0.1:2379,http://127.0.0.1:4001        - -advertise-client-urls        - http://127.0.0.1:2379,http://127.0.0.1:4001        - -initial-cluster-token        - skydns-etcd        volumeMounts:        - mountPath: /home/data/etcd          name: etcd-storage      - name: kube2sky        #image: gcr.io/google_containers/kube2sky:1.11        image: 192.168.100.90:5000/duni/kube2sky:1.14        imagePullPolicy: IfNotPresent        resources:          limits:            cpu: 100m            memory: 50Mi        livenessProbe:          httpGet:            path: /healthz            port: 8080            scheme: HTTP          initialDelaySeconds: 60          timeoutSeconds: 5          successThreshold: 1          failureThreshold: 5        readinessProbe:          httpGet:            path: /readiness            port: 8081            scheme: HTTP          initialDelaySeconds: 30          timeoutSeconds: 5        args:        # command = "/kube2sky"       # - -etcd-server=http://127.0.0.1:4001        #- -kube_master_url=http://172.27.8.210:8080        - --kube-master-url=http://192.168.100.27:8080        - --domain=cluster.local      - name: skydns        #image: gcr.io/google_containers/skydns:2015-03-11-001        image: 192.168.100.90:5000/duni/skydns:2015-10-13-8c72f8c        imagePullPolicy: IfNotPresent        resources:          limits:            cpu: 100m            memory: 50Mi        args:        # command = "/skydns"        - -machines=http://127.0.0.1:2379        - -addr=0.0.0.0:53        - -ns-rotate=false        - -domain=cluster.local        ports:        - containerPort: 53          name: dns          protocol: UDP        - containerPort: 53          name: dns-tcp          protocol: TCP        livenessProbe:          httpGet:            path: /healthz            port: 8080            scheme: HTTP          initialDelaySeconds: 30          timeoutSeconds: 5        readinessProbe:          httpGet:            path: /healthz            port: 8080            scheme: HTTP          initialDelaySeconds: 1          timeoutSeconds: 5      - name: healthz        #image: gcr.io/google_containers/exechealthz:1.0        image: 192.168.100.90:5000/duni/exechealthz-amd64:latest        imagePullPolicy: IfNotPresent        resources:          limits:            cpu: 10m            memory: 20Mi        args:        - -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null        - -port=8080        ports:        - containerPort: 8080          protocol: TCP      volumes:      - name: etcd-storage        emptyDir: {}      dnsPolicy: Default  # Don't use cluster DNS.

skydns-rc.yaml文件中需要根据自己的实际情况进行相关修改

1、把镜像改为自己私有仓库路径

$ cat skydns-rc.yaml | grep imageimage: 192.168.100.90:5000/duni/etcd-amd64:3.0.17imagePullPolicy: IfNotPresent#image: gcr.io/google_containers/kube2sky:1.11image: 192.168.100.90:5000/duni/kube2sky:1.14imagePullPolicy: IfNotPresent#image: gcr.io/google_containers/skydns:2015-03-11-001image: 192.168.100.90:5000/duni/skydns:2015-10-13-8c72f8cimagePullPolicy: IfNotPresent#image: gcr.io/google_containers/exechealthz:1.0image: 192.168.100.90:5000/duni/exechealthz-amd64:latestimagePullPolicy: IfNotPresent

建议镜像到阿里云容器镜像中查找，然后更改镜像tag，再push到自己搭建的私有仓库中，如何搭建自己的私有仓库

2、kube2sky容器中参数：- --kube-master-url=http://192.168.100.27:8080设为你自己的k8s集群master主机ip:port, - --domain=cluster.local 设置集群中service域名（可自行定义一个名字）

3、skydns容器中参数：- -domain=cluster.local，需跟kube2sky中设置的名称一致

二、skydns-svc.yaml 文件

apiVersion: v1kind: Servicemetadata:  name: kube-dns  namespace: default  labels:    k8s-app: kube-dns    kubernetes.io/cluster-service: "true"    kubernetes.io/name: "KubeDNS"spec:  selector:    k8s-app: kube-dns  clusterIP:  10.254.0.100  ports:  - name: dns    port: 53    protocol: UDP  - name: dns-tcp    port: 53    protocol: TCP

此处可自行修改clusterIP: 10.254.0.100 ip

三、修改k8s集群中节点机的kubelet启动参数

$ vi /etc/kubernetes/kubelet    # centos系统，ubuntu系统在不同路径下KUBELET_ARGS="--cluster_dns=10.254.0.100 --cluster_domain=cluster.local"

注：--cluster_dns必须跟skydns-svc.yaml文件中的clusterIP值相同，cluster_domain必须跟skydns-rc.yaml文件中skydns和kube2sky容器中的domain参数值相同

重启kubelet

systemctl restart kubelet

四、创建dns pod和service

kubectl create -f skydns-rc.yaml
kubectl create -f skydns-svc.yaml

查看pod和service状态

$ kubectl get pods --all-namespaces | grep kube-dns    NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGEdefault       kube-dns-v9-vldgj                       4/4       Running   0          3h$ kubectl get pods --all-namespaces | grep kube-dns    NAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGEdefault       kube-dns               10.254.0.100     <none>        53/UDP,53/TCP    3h

五、验证dns

busybox.yaml

apiVersion: v1kind: Podmetadata:  name: busybox  namespace: defaultspec:  containers:  - image: busybox    command:      - sleep      - "3600"    imagePullPolicy: IfNotPresent    name: busybox  restartPolicy: Always

创建busybox pod

$ kubectl create -f busybox.yaml$ kubectl get pods --all-namespaces | grep busyboxNAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGEdefault       busybox                                 1/1       Running   3          3h

验证dns解析

# 查看所有的service（以下是我master主机所有服务）$ kbuectl get svc --all-namespacesNAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGEdefault       frontend               10.254.204.132   <nodes>       80:30001/TCP     19hdefault       kube-dns               10.254.0.100     <none>        53/UDP,53/TCP    3hdefault       kubernetes             10.254.0.1       <none>        443/TCP          30ddefault       mysql-service          10.254.246.96    <nodes>       3306:30365/TCP   21hdefault       redis-master           10.254.172.30    <none>        6379/TCP         19hdefault       redis-service          10.254.253.213   <none>        6379/TCP         19hkube-system   heapster               10.254.145.32    <none>        80/TCP           18hkube-system   kubernetes-dashboard   10.254.163.216   <nodes>       80:30009/TCP     20hkube-system   monitoring-grafana     10.254.199.203   <none>        80/TCP           18hkube-system   monitoring-influxdb    10.254.27.3      <none>        8086/TCP         18h# 验证$ kubectl exec -it busybox nslookup kubernetesServer:    10.254.0.100Address 1: 10.254.0.100nslookup: can't resolve 'kubernetes'$ kubectl exec -it busybox nslookup kubernetes.defaultServer:    10.254.0.100Address 1: 10.254.0.100nslookup: can't resolve 'kubernetes.default': Try again$ kubectl exec -it busybox nslookup www.baidu.comServer:    10.254.0.100Address 1: 10.254.0.100Name:      www.baidu.comAddress 1: 14.215.177.38Address 2: 14.215.177.37

好吧，所有pod运行正常，可dns就是不能根据本地域名解析其ip，发现解析百度，网易等却可以

六、查找dns不能根据域名解析其ip的原因

查看dns pod中各个容器的日志，查看skydns容器时，发现以下错误，请求超时

    $ kubectl logs --namespace=default $(kubectl get pods --namespace=default -l k8s-app=kube-dns -o name) -c skydns    2017-04-26T07:15:35.141855000Z 2017/04/26 07:15:35 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"    2017-04-26T07:18:09.141845000Z 2017/04/26 07:18:09 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"    2017-04-26T07:21:53.045513000Z 2017/04/26 07:21:53 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"    2017-04-26T07:26:13.142510000Z 2017/04/26 07:26:13 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"    2017-04-26T07:28:25.045739000Z 2017/04/26 07:28:25 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"

于是谷歌，找到原因所在，192.168.100.1 这个dns服务地址并不可用，那我们就用谷歌公开的dns

修改集群master主机，新增谷歌dns

$ vi /etc/resolv.conf # Generated by NetworkManagernameserver 192.168.100.1# 新增下面两行nameserver 8.8.8.8nameserver 8.8.4.4

修改skydns-rc.yaml 中 skydns 容器的args参数

args:# command = "/skydns"- -machines=http://127.0.0.1:2379- -addr=0.0.0.0:53- -ns-rotate=false- -domain=cluster.local- -nameservers=8.8.8.8:53,8.8.4.4:53     # 新增这行

重启启动dns 和 busybox pod

$ kubectl delete rc kube-dns --namespace=default$ kubectl delete src kube-dns --namespace=default$ kubectl delete pods busybox --namespace=default$ kubectl create -f skydns-rc.yaml$ kubectl create -f skydns-svc.yaml$ kubectl create -f busybox.yaml

七、再次验证dns

$ kubectl exec -it busybox nslookup kubernetesServer:    10.254.0.100Address 1: 10.254.0.100Name:      kubernetesAddress 1: 10.254.0.1$ kubectl exec -it busybox nslookup kubernetes.defaultServer:    10.254.0.100Address 1: 10.254.0.100Name:      kubernetes.defaultAddress 1: 10.254.0.1$ kubectl exec -it busybox nslookup heapsterServer:    10.254.0.100Address 1: 10.254.0.100nslookup: can't resolve 'heapster'$ kubectl exec -it busybox nslookup heapster.kube-systemServer:    10.254.0.100Address 1: 10.254.0.100Name:      heapster.kube-systemAddress 1: 10.254.145.32

注：我们的dns pod所在的命名空间（namespace）是在default，当我们需要查询的服务跟dns pod不在同一命名空间时，需通过域名.命名空间 方式进行查找，如上面的heapster.kube-system

友情提示：一定要学会查看pod中的容器日志，因为很多时候，我们一个pod中某个容器只要一个参数书写错误，则该容器就running失败了

kubectl logs –namespace=namespace_name $(kubectl get pods –namespace=namespace_name -l label_name=label_value -o name) -c container_name

如上面查找dns pod中skydns容器日志

kubectl logs –namespace=default $(kubectl get pods –namespace=default -l k8s-app=kube-dns -o name) -c skydns