《kubernetes-1.8.0》12-addon-ingress-nginx

《kubernetes 1.8.0 测试环境安装部署》

时间：2017-11-29

一、ingress介绍

关于nginx-ingress：

官网上摘抄几句：

Typically, services and pods have IPs only routable by the cluster network. All traffic that ends up at an edge router is either dropped or forwarded elsewhere. Conceptually, this might look like:
通常，services和Pod只能通过集群内网络访问。 所有在边界路由器上的流量都被丢弃或转发到别处。 从概念上讲，这可能看起来像：

    internet        |  ------------  [ Services ]

An Ingress is a collection of rules that allow inbound connections to reach the cluster services.
Ingress是允许入站连接到达集群服务的一组规则。即介于物理网络和群集svc之间的一组转发规则。
其实就是实现L4 L7的负载均衡

    internet        |   [ Ingress ]   --|-----|--   [ Services ]

出来ingress以外，对外的服务发现还可以使用nodeport的方式，但这样会导致节点上的端口占用很多，且不利于管理。

It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting, and more. Users request ingress by POSTing the Ingress resource to the API server. An Ingress controller is responsible for fulfilling the Ingress, usually with a loadbalancer, though it may also configure your edge router or additional frontends to help handle the traffic in an HA manner.
它可以配置为提供外部可访问的URL，负载平衡流量，终止SSL，提供基于名称的虚拟主机，等等。 用户请求通过将Ingress resource发送到API server。 一个Ingress controller负责响应入口请求，通常是负载均衡方式，它也可能通过配置你的外部路由器或额外的前端，以帮助处理HA方式的流量。

二、ingress-nginx部署

ingress-nginx组件有几个部分组成：

• configmap.yaml:提供configmap可以在线更行nginx的配置
• default-backend.yaml:提供一个缺省的后台错误页面 404
• namespace.yaml:创建一个独立的命名空间 ingress-nginx
• rbac.yaml:创建对应的role rolebinding 用于rbac
• tcp-services-configmap.yaml:修改L4负载均衡配置的configmap
• udp-services-configmap.yaml:修改L4负载均衡配置的configmap
• with-rbac.yaml:有应用rbac的nginx-ingress-controller组件

相关yaml文件如下：

namespace.yaml

apiVersion: v1kind: Namespacemetadata:  name: ingress-nginx

default-backend.yaml

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: default-http-backend  labels:    app: default-http-backend  namespace: ingress-nginxspec:  replicas: 1  template:    metadata:      labels:        app: default-http-backend    spec:      terminationGracePeriodSeconds: 60      containers:      - name: default-http-backend        # Any image is permissable as long as:        # 1. It serves a 404 page at /        # 2. It serves 200 on a /healthz endpoint        image: gcr.mirrors.ustc.edu.cn/google_containers/defaultbackend:1.4        livenessProbe:          httpGet:            path: /healthz            port: 8080            scheme: HTTP          initialDelaySeconds: 30          timeoutSeconds: 5        ports:        - containerPort: 8080        resources:          limits:            cpu: 10m            memory: 20Mi          requests:            cpu: 10m            memory: 20Mi---apiVersion: v1kind: Servicemetadata:  name: default-http-backend  namespace: ingress-nginx  labels:    app: default-http-backendspec:  ports:  - port: 80    targetPort: 8080  selector:    app: default-http-backend

• github上 image: gcr.io/google_containers/defaultbackend:1.4 更换为 gcr.mirrors.ustc.edu.cn/google_containers/defaultbackend:1.4

configmap.yaml

kind: ConfigMapapiVersion: v1metadata:  name: nginx-configuration  namespace: ingress-nginx  labels:    app: ingress-nginx

tcp-services-configmap.yaml

apiVersion: v1metadata:  name: tcp-services  namespace: ingress-nginx

udp-services-configmap.yaml

kind: ConfigMapapiVersion: v1metadata:  name: udp-services  namespace: ingress-nginx

rbac.yaml

apiVersion: v1kind: ServiceAccountmetadata:  name: nginx-ingress-serviceaccount  namespace: ingress-nginx---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRolemetadata:  name: nginx-ingress-clusterrolerules:  - apiGroups:      - ""    resources:      - configmaps      - endpoints      - nodes      - pods      - secrets    verbs:      - list      - watch  - apiGroups:      - ""    resources:      - nodes    verbs:      - get  - apiGroups:      - ""    resources:      - services    verbs:      - get      - list      - watch  - apiGroups:      - "extensions"    resources:      - ingresses    verbs:      - get      - list      - watch  - apiGroups:      - ""    resources:        - events    verbs:        - create        - patch  - apiGroups:      - "extensions"    resources:      - ingresses/status    verbs:      - update---apiVersion: rbac.authorization.k8s.io/v1beta1kind: Rolemetadata:  name: nginx-ingress-role  namespace: ingress-nginxrules:  - apiGroups:      - ""    resources:      - configmaps      - pods      - secrets      - namespaces    verbs:      - get  - apiGroups:      - ""    resources:      - configmaps    resourceNames:      # Defaults to "<election-id>-<ingress-class>"      # Here: "<ingress-controller-leader>-<nginx>"      # This has to be adapted if you change either parameter      # when launching the nginx-ingress-controller.      - "ingress-controller-leader-nginx"    verbs:      - get      - update  - apiGroups:      - ""    resources:      - configmaps    verbs:      - create  - apiGroups:      - ""    resources:      - endpoints    verbs:      - get---apiVersion: rbac.authorization.k8s.io/v1beta1kind: RoleBindingmetadata:  name: nginx-ingress-role-nisa-binding  namespace: ingress-nginxroleRef:  apiGroup: rbac.authorization.k8s.io  kind: Role  name: nginx-ingress-rolesubjects:  - kind: ServiceAccount    name: nginx-ingress-serviceaccount    namespace: ingress-nginx---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata:  name: nginx-ingress-clusterrole-nisa-bindingroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: nginx-ingress-clusterrolesubjects:  - kind: ServiceAccount    name: nginx-ingress-serviceaccount    namespace: ingress-nginx

with-rbac.yaml

apiVersion: extensions/v1beta1kind: DaemonSetmetadata:  name: nginx-ingress-controller  namespace: ingress-nginx spec:  selector:    matchLabels:      app: ingress-nginx  template:    metadata:      labels:        app: ingress-nginx      annotations:        prometheus.io/port: '10254'        prometheus.io/scrape: 'true'    spec:      serviceAccountName: nginx-ingress-serviceaccount      hostNetwork: true      containers:        - name: nginx-ingress-controller          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.19          args:            - /nginx-ingress-controller            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend            - --configmap=$(POD_NAMESPACE)/nginx-configuration            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services            - --annotations-prefix=nginx.ingress.kubernetes.io          env:            - name: POD_NAME              valueFrom:                fieldRef:                  fieldPath: metadata.name            - name: POD_NAMESPACE              valueFrom:                fieldRef:                  fieldPath: metadata.namespace          ports:          - name: http            containerPort: 80          - name: https            containerPort: 443          livenessProbe:            failureThreshold: 3            httpGet:              path: /healthz              port: 10254              scheme: HTTP            initialDelaySeconds: 10            periodSeconds: 10            successThreshold: 1            timeoutSeconds: 1          readinessProbe:            failureThreshold: 3            httpGet:              path: /healthz              port: 10254              scheme: HTTP            periodSeconds: 10            successThreshold: 1            timeoutSeconds: 1      nodeSelector:        custom/ingress-controller-ready: "true"

这里做了一些改动，如下：

• kind: DaemonSet：官档给的是deployment，replicate 为 1，这样将会在某一台节点上启动对应的nginx-ingress-controller pod。外部流量访问至该节点，由该节点负载分担至内部的service。测试环境考虑防止单点故障，改为DaemonSet，配合亲和性部署在制定节点上启动nginx-ingress-controller pod，确保有多个节点启动nginx-ingress-controller pod，后续将这些节点加入到外部硬件负载均衡组实现高可用性。
• hostNetwork: true：添加该字段，暴露nginx-ingress-controller pod的服务端口（80）
• nodeSelector: 加入亲和性部署，有custom/ingress-controller-ready label的才会部署该DaemonSet

测试环境中将三台master打上label

kubectl label nodes node.131 custom/ingress-controller-ready=truekubectl label nodes node.132 custom/ingress-controller-ready=truekubectl label nodes node.133 custom/ingress-controller-ready=true

加载对应yaml文件

kubectl create -f namespace.yamlkubectl create -f default-backend.yamlkubectl create -f configmap.yamlkubectl create -f tcp-services-configmap.yamlkubectl create -f udp-services-configmap.yamlkubectl create -f rbac.yamlkubectl create -f with-rbac.yaml

查看pod创建情况：

[root@node-131 deploy]#  kubectl get pods --all-namespaces -l app=ingress-nginx --watchNAMESPACE       NAME                             READY     STATUS    RESTARTS   AGEingress-nginx   nginx-ingress-controller-5jxp5   1/1       Running   0          1hingress-nginx   nginx-ingress-controller-845z5   1/1       Running   0          1hingress-nginx   nginx-ingress-controller-wprgh   1/1       Running   0          1h都running了就可以ctrl-c了

三、测试ingress

##查看default命名空间里的svc：[root@node-131 ~]# kubectl get svcNAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGEdefault-http-backend   NodePort    10.254.196.136   <none>        80:45135/TCP   1ddemo-service           NodePort    10.254.23.220    <none>        80:31001/TCP   5dkubernetes             ClusterIP   10.254.0.1       <none>        443/TCP        6dnginx-service          NodePort    10.254.197.49    <none>        80:31000/TCP   5d##之前案例中留下了 demo-service  、 nginx-service，直接拿来做测试：

通过域名进行url转发，对应yaml：

demo-ingress.yaml

apiVersion: extensions/v1beta1kind: Ingressmetadata:  name: demo-ingress  namespace: defaultspec:  rules:  - host: demo.bs.ingress    http:      paths:      - path: /        backend:          serviceName: demo-service          servicePort: 80  - host: nginx.bs.ingress    http:      paths:      - path: /        backend:          serviceName: nginx-service          servicePort: 80

• host: 对应的域名
• path: url上下文
• backend:后向转发 到对应的 serviceName: servicePort:

应用yaml文件：

kubectl apply -f demo-ingress.yaml ##查看ingress resource：[root@node-131 ingress]# kubectl get ingNAME           HOSTS                              ADDRESS   PORTS     AGEdemo-ingress   demo.bs.ingress,nginx.bs.ingress             80        5h

修改hosts文件并测试访问：

echo '172.18.169.131 demo.bs.ingress' >> /etc/hostsecho '172.18.169.132 nginx.bs.ingress' >> /etc/hosts###ip为启动nginx-controller的节点##测试访问：[root@node-131 ingress]# curl nginx.bs.ingress<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>    body {        width: 35em;        margin: 0 auto;        font-family: Tahoma, Verdana, Arial, sans-serif;    }</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed andworking. Further configuration is required.</p><p>For online documentation and support please refer to<a href="http://nginx.org/">nginx.org</a>.<br/>Commercial support is available at<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p></body></html>------[root@node-131 ingress]# curl demo.bs.ingress<!DOCTYPE html><html lang="zh"><head>    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">    <title>Running!</title>    <style type="text/css">        body {            width: 100%;            min-height: 100%;            background: linear-gradient(to bottom, #fff 0, #b8edff 50%, #83dfff 100%);            background-attachment: fixed;        }    </style></head><body class=" hasGoogleVoiceExt"><div align="center">    <h1>Your container is running!</h1>    <img src="./docker.png" alt="docker"></div></body></html>

更多ingress写法，查看官方文档：ingress

本系列其他内容：

• 01-环境准备

• 02-etcd群集搭建

• 03-kubectl管理工具

• 04-master搭建

• 05-node节点搭建

• 06-addon-calico

• 07-addon-kubedns

• 08-addon-dashboard

• 09-addon-kube-prometheus

• 10-addon-EFK

• 11-addon-Harbor

• 12-addon-ingress-nginx

• 13-addon-traefik

参考资料

https://github.com/kubernetes/ingress-nginx/tree/master/deploy

https://kubernetes.io/docs/concepts/services-networking/ingress/