kubernetes学习记录（10）——建立Heapster+Influxdb+Grafana集群性能监控平台

采用的是Heapster+Influxdb+Grafana建立集群性能监控平台。

据说Heapster需要与Kubernetes Master进行安全连接，所以需要对集群进行安全认证，我的集群环境已经进行了安全认证。

非安全认证的集群能否使用Heapster，我没有验证。

集群的安全认证可以参考我的博客kubernetes学习记录（9）——集群基于CA签名的安全设置（可能有坑，这块还没研究的特别明白，网上的各种认证方式都有，我是综合参考的，不一定完美）

参考博客Kubernetes heapster监控插件安装文档与在开启TLS的Kubernetes1.6集群上安装heapster进行整理。

从作者分享的地址下载所需的镜像文件，Push到自己的本地镜像仓库中。

index.tenxcloud.com/jimmy/heapster-amd64:v1.3.0-beta.1index.tenxcloud.com/jimmy/heapster-influxdb-amd64:v1.1.1index.tenxcloud.com/jimmy/heapster-grafana-amd64:v4.0.2

yaml源码来源自kubernetes GitHub。

安装Heapster

heapster-deployment.yaml

修改- --source为自己的master apiserver访问地址
修改image地址

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: heapster  namespace: kube-systemspec:  replicas: 1  template:    metadata:      labels:        task: monitoring        k8s-app: heapster    spec:      containers:      - name: heapster        image: 192.168.121.140:5000/heapster-amd64        imagePullPolicy: IfNotPresent        command:        - /heapster        - --source=kubernetes:http://192.168.121.143:8080        - --sink=influxdb:http://monitoring-influxdb:8086        - --metric_resolution=60s

heapster-service.yaml

apiVersion: v1kind: Servicemetadata:  labels:    task: monitoring    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)    # If you are NOT using this as an addon, you should comment out this line.    kubernetes.io/cluster-service: 'true'    kubernetes.io/name: Heapster  name: heapster  namespace: kube-systemspec:  ports:  - port: 80    targetPort: 8082  selector:    k8s-app: heapster

创建deployment和service

#kubectl create -f heapster-deployment.yaml#kubectl create -f heapster-service.yaml

安装Influxdb

influxdb 官方建议使用命令行或 HTTP API 接口来查询数据库，从 v1.1.0 版本开始默认关闭 admin UI，将在后续版本中移除 admin UI 插件。

开启镜像中 admin UI的办法如下：先导出镜像中的 influxdb 配置文件，开启插件后，再将配置文件内容写入 ConfigMap，最后挂载到镜像中，达到覆盖原始配置的目的。

$ #在镜像所在的宿主机上，导出镜像中的influxdb配置文件$ docker run --rm --entrypoint 'cat'  -ti heapster-influxdb-amd64:v1.1.1 /etc/config.toml >config.toml.orig$ cp config.toml.orig config.toml$ # 修改：启用 admin 接口$ vim config.toml修改第35行<   enabled = false--->   enabled = true$ #将修改后的config.toml拷贝到Master上，再将修改后的配置写入到ConfigMap对象中$ kubectl create configmap influxdb-config --from-file=config.toml -n kube-system$ # 将ConfigMap中的配置文件挂载到Pod中，达到覆盖原始配置的目的

最终的influxdb-deployment.yaml文件如下：

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: monitoring-influxdb  namespace: kube-systemspec:  replicas: 1  template:    metadata:      labels:        task: monitoring        k8s-app: influxdb    spec:      containers:      - name: influxdb        image: 192.168.121.140:5000/heapster-influxdb-amd64        volumeMounts:        - mountPath: /data          name: influxdb-storage        - mountPath: /etc/          name: influxdb-config      volumes:      - name: influxdb-config        configMap:          name: influxdb-config      - name: influxdb-storage        emptyDir: {}

influxdb-service.yaml

apiVersion: v1kind: Servicemetadata:  labels:    task: monitoring    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)    # If you are NOT using this as an addon, you should comment out this line.    kubernetes.io/cluster-service: 'true'    kubernetes.io/name: monitoring-influxdb  name: monitoring-influxdb  namespace: kube-systemspec:  type: NodePort  ports:  - port: 8086    targetPort: 8086    name: http  - port: 8083    targetPort: 8083    name: api  selector:    k8s-app: influxdb

创建deployment和service

#kubectl create -f influxdb-deployment.yaml#kubectl create -f influxdb-service.yaml

安装grafana

grafana-deployment.yaml
修改GF_SERVER_ROOT_URL的value

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: monitoring-grafana  namespace: kube-systemspec:  replicas: 1  template:    metadata:      labels:        task: monitoring        k8s-app: grafana    spec:      containers:      - name: grafana        image: 192.168.121.140:5000/heapster-grafana-amd64        ports:          - containerPort: 3000            protocol: TCP         volumeMounts:        - mountPath: /var          name: grafana-storage        env:        - name: INFLUXDB_HOST          value: monitoring-influxdb        - name: GRAFANA_PORT          value: "3000"          # The following env variables are required to make Grafana accessible via          # the kubernetes api-server proxy. On production clusters, we recommend          # removing these env variables, setup auth for grafana, and expose the grafana          # service using a LoadBalancer or a public IP.        - name: GF_AUTH_BASIC_ENABLED          value: "false"        - name: GF_AUTH_ANONYMOUS_ENABLED          value: "true"        - name: GF_AUTH_ANONYMOUS_ORG_ROLE          value: Admin        - name: GF_SERVER_ROOT_URL          # If you're only using the API Server proxy, set this value instead:          value: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/          #value: /      volumes:      - name: grafana-storage        emptyDir: {}

grafana-service.yaml

apiVersion: v1kind: Servicemetadata:  labels:    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)    # If you are NOT using this as an addon, you should comment out this line.    kubernetes.io/cluster-service: 'true'    kubernetes.io/name: monitoring-grafana  name: monitoring-grafana  namespace: kube-systemspec:  # In a production setup, we recommend accessing Grafana through an external Loadbalancer  # or through a public IP.  # type: LoadBalancer  # You could also use NodePort to expose the service at a randomly-generated port  # type: NodePort  ports:  - port: 80    targetPort: 3000  selector:    k8s-app: grafana

创建deployment和service

#kubectl create -f grafana-deployment.yaml#kubectl create -f grafana-service.yaml

访问验证

验证Heapster

访问kubernets dashboard （masterIP:8080/ui）界面，看是显示各 Nodes、Pods 的 CPU、内存、负载等利用率曲线图。

验证Influxdb

获取 influxdb http 8086 映射的 NodePort

#kubectl get svc -n kube-system|grep influxdb

8086对应的端口是32450。

通过 kube-apiserver 的非安全端口访问 influxdb 的 admin UI 界面：

http://masterIP:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb:8083/

Host 中输入 Influxdb pod所在的node IP， Port 中输入 8086 映射的 nodePort 如上面的 32450，点击 “Save” 即可

回车

验证Grafana

获取 grafana 服务 URL

#kubectl cluster-info

我的集群安全认证还有一些小细节上的问题，这里不应该显示localhost的。
替换成MasterIP。
grafana 服务 URL：
http://192.168.121.143:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana