Kubernetes部署大数据组件系列一：一键部署Zookeeper集群
来源：互联网 发布：电脑免费发信息软件 编辑：程序博客网 时间：2024/06/07 17:19
目前的大数据平台存在的问题：
- 通过手工方式直接部署在物理机上，过程复杂、效率低，不适合大规模集群的部署和动态扩容。
- 由于平台组件繁多，对宿主机底层库依赖过高，一旦安装完成，很难进行迁移，也无法做成模版进行重复使用。
- 组件升级不够灵活，往往会对有相同底层依赖的其他组件带来影响。
- 采用虚机部署时，会对宿主机产生资源浪费、开销大。
- 物理机或虚机关启等管理操作速度较慢。
- 无法在同一套物理集群上部署多套大数据平台，使用虚机的方案太耗费资源。
针对上述问题，我们打算用时下流行的Docker容器来封装大数据各个组件，用Google的神器Kubernetes来编排这些Docker容器，最终达到：
- 平台能够自动化的快速部署（小规模集群能够达到秒级）、动态扩容。
- 平台构建一次，就能够形成模版重复使用、灵活迁移。
- 平台构建一次，就能够灵活部署在不同的环境中（比如公有云、私有云、裸机，虚拟机）。
- 组件能够灵活升级、灵活替代。
- 消除线上线下环境不一致导致的问题，再也不会有“在我的机器上运行的好好的”这种托词。
- 因为其多资源的低消耗，在物理集群上能同时部署多套大数据平台，供不同部门独 享。
- 很适合持续集成/持续部署(CI/CD)的环境，因为可以自动的下载、绿色的安装和彻底清除应用程序，不会影响宿主机的环境。
本篇是该系列的第一篇，手把手教你通过K8s部署zookeeper集群，涉及到的概念，各位看官自行脑补，我只讲一下部署过程，直接上干货了。
- 使用Kargo搭建Kubernetes 1.6集群:
地址 角色 10.0.7.14Ansible Client （Kargo用到的工具）10.0.8.182Master, NODE 110.0.8.183NODE 2[下文提到的kargo.tg、kargo_images_v1.6.1.tar.gz压缩包]http://pan.baidu.com/s/1kUKyfBL
1，我用笔记本（macOS系统），作为Ansible client。先新建一个文件夹kargo，把kargo.tgz放进去并解压，目录结构如下：
2，需要安装python 2.7、pip 9.0.1、python-netaddr、ansible 2.3.1.0等工具[ansible安装]http://valdhaus.co/writings/ansible-mac-osx/
3，修改 inventory/inventory.cfg
wangliangdeMacBook-Pro:kargo wangliang$ pwd/Users/wangliang/kargo/kargowangliangdeMacBook-Pro:kargo wangliang$ vim inventory/inventory.cfg[all]node1 ansible_user=app ansible_host=10.0.8.182 ip=10.0.8.182 access_ip=10.0.8.182node2 ansible_user=app ansible_host=10.0.8.183 ip=10.0.8.183 access_ip=10.0.8.183[kube-master]node1 ip=10.0.8.182 access_ip=10.0.8.182[kube-node]node1node2[etcd]node1[k8s-cluster:children]kube-nodekube-master[calico-rr]4，分别在10.0.8.182(Node 1)、10.0.8.183(Node 2)上做如下操作：
首先安装Docker 1.13.1，并启动服务
[Ubuntu安装Docker]http://www.linuxidc.com/Linux/2017-01/139985.htm其次把kargo_images_v1.6.1.tar.gz 拷贝到要安装k8s的节点上，放到k8s文件夹内，并解压，目录结构如下：
5，在kargo_images_v1.6.1目录下，进入bash，执行：➜ k8s cd kargo_images_v1.6.1➜ kargo_images_v1.6.1 pwd/data/k8s/kargo_images_v1.6.1➜ kargo_images_v1.6.1 bashapp@yuyan1:/data/k8s/kargo_images_v1.6.1$images=$(ls -l ../kargo_images_v1.6.1|awk -F' ' '{ print $9 }')app@yuyan1:/data/k8s/kargo_images_v1.6.1$ for x in ${images[*]}; do sudo docker load -i $x; done将目录下的镜像放入docker images。
6， Node1、Node2和Mac间都要SSH免密，Mac上还要打开共享里面的允许远端访问
Node1、Node2、Mac都在用户目录下执行➜ ~ cd /home/appssh-keygen -t rsa #一路回车就行Generating public/private rsa key pair.Enter file in which to save the key (/home/app/.ssh/id_rsa):Created directory '/home/app/.ssh'.Enter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /home/app/.ssh/id_rsa.Your public key has been saved in /home/app/.ssh/id_rsa.pub.The key fingerprint is:7c:f4:2f:21:f2:85:7f:20:38:02:1d:ef:79:39:b6:be root@dcos1The key's randomart image is:+--[ RSA 2048]----+| . || . o || . . . . || . o + + || . S X = || . B B + || o o o || . o || E. |+-----------------+➜ ~ cd /home/app/.ssh➜ .ssh ssh-copy-id Node2➜ .ssh ssh-copy-id Mac每次执行，步骤都如下类似：The authenticity of host 'Node1 (10.0.8.183)' can't be established.ECDSA key fingerprint is 91:1a:13:a8:57:2b:a0:42:4d:aa:c9:83:c3:33:16:f9.Are you sure you want to continue connecting (yes/no)? yes/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keysroot@dcos1's password:Number of key(s) added: 1Now try logging into the machine, with: "ssh 'Node2'"and check to make sure that only the key(s) you wanted were added.所有虚机都执行完成后，可以通过ssh hostname来查看之间是否ssh连接已经免密了，如下测试
➜ ~ ssh 10.0.8.183Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-62-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage180 packages can be updated.75 updates are security updates.Last login: Wed Jun 28 09:16:54 2017 from 10.0.7.14➜ ~7，在Ansible client(Mac)上执行
$ ansible-playbook -i ~/kargo/inventory/inventory.cfg cluster.yml -b -v --user=username --ask-sudo-pass安装成功，在Mac上可以看到如下输出：
在Node 1上可以看到：
遇到的错误：
1，有如下错误：kubectl The connection to the server localhost:8080 was refused修改了vim inventory/group_vars/k8s-cluster.ym 里面的8080为28080，避免和别人冲突，又有如下错误：
kubernetes/master : Master | wait for the apiserver to be running原因是apiserver的镜像起不了，无法提供服务
怀疑可能是不支持docker 1.17版本，所以更换为docker 1.13.1版本子再次安装后没有上述错误了，安装成功。切记，要用docker 1.13.1版本。2，如果按照上述操作始终没有成功，可以把节点上的
/run/kubernetes/etc/kubernetes都删除，然后把docker也都卸载重新装。然后在Ansible client节点上重新部署就可以啦。
- 使用Dockerfile制作包含zkui 2.0 的Zookeeper 3.4.10版本的镜像：
1，建立你要做docker的目录，结构如下：2，其中Dockerfile如下：
FROM openjdk:8-jre-alpineMAINTAINER Wang Liang <wangl8@knownsec.com># Install required packagesRUN apk add --no-cache \ bash \ su-execENV ZOO_USER=zookeeper \ ZOO_CONF_DIR=/conf \ ZOO_DATA_DIR=/data \ ZOO_UI_DIR=/zkui \ ZOO_DATA_LOG_DIR=/datalog \ ZOO_PORT=2181 \ ZOO_TICK_TIME=2000 \ ZOO_INIT_LIMIT=5 \ ZOO_SYNC_LIMIT=2# Add a user and make dirsRUN set -x \ && adduser -D "$ZOO_USER" \ && mkdir -p "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR" \ && chown "$ZOO_USER:$ZOO_USER" "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR"ARG GPG_KEY=C823E3E5B12AF29C67F81976F5CECB3CB5E9BD2DARG DISTRO_NAME=zookeeper-3.4.10# Download Apache Zookeeper, verify its PGP signature, untar and clean upRUN set -x \ && apk add --no-cache --virtual .build-deps \ gnupg \ && wget -q "http://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz" \ && wget -q "http://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz.asc" \ && export GNUPGHOME="$(mktemp -d)" \ && gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" \ && gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz" \ && tar -xzf "$DISTRO_NAME.tar.gz" \ && mv "$DISTRO_NAME/conf/"* "$ZOO_CONF_DIR" \ && rm -r "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc" \ && apk del .build-depsADD zkui-master $ZOO_UI_DIRWORKDIR $DISTRO_NAMEVOLUME ["$ZOO_DATA_DIR", "$ZOO_DATA_LOG_DIR"]EXPOSE $ZOO_PORT 2888 3888ENV PATH=$PATH:/$DISTRO_NAME/bin:$ZOO_UI_DIR \ ZOOCFGDIR=$ZOO_CONF_DIRCOPY docker-entrypoint.sh /ENTRYPOINT ["/docker-entrypoint.sh"]3，docker-entrypoint.sh如下：
#!/bin/bashset -e# Allow the container to be started with `--user`if [ "$1" = 'zkServer.sh' -a "$(id -u)" = '0' ]; then chown -R "$ZOO_USER" "$ZOO_DATA_DIR" "$ZOO_DATA_LOG_DIR" exec su-exec "$ZOO_USER" "$0" "$@"fi# Generate the config only if it doesn't existif [ ! -f "$ZOO_CONF_DIR/zoo.cfg" ]; then CONFIG="$ZOO_CONF_DIR/zoo.cfg" echo "clientPort=$ZOO_PORT" >> "$CONFIG" echo "dataDir=$ZOO_DATA_DIR" >> "$CONFIG" echo "dataLogDir=$ZOO_DATA_LOG_DIR" >> "$CONFIG" echo "tickTime=$ZOO_TICK_TIME" >> "$CONFIG" echo "initLimit=$ZOO_INIT_LIMIT" >> "$CONFIG" echo "syncLimit=$ZOO_SYNC_LIMIT" >> "$CONFIG" for server in $ZOO_SERVERS; do echo "$server" >> "$CONFIG" donefiif [ -f "$ZOO_UI_DIR/config.cfg" ]; then CONFIG="$ZOO_UI_DIR/config.cfg" echo "serverPort=$ZOO_UI_PORT" >> "$CONFIG" echo "zkServer=$ZOO_UI_SERVER" >> "$CONFIG"fi# Write myid only if it doesn't existif [ ! -f "$ZOO_DATA_DIR/myid" ]; then echo "${ZOO_MY_ID:-1}" > "$ZOO_DATA_DIR/myid"ficd $ZOO_UI_DIRexec nohup java -jar zkui-2.0-SNAPSHOT-jar-with-dependencies.jar &exec zkServer.sh start-foreground[制作过程需要的文件]http://pan.baidu.com/s/1i5IG6sH
执行
➜ wldocker sudo docker build -t zookeeper_3.4.10_zkui_2.0:0.0.1 .就可以制作出包含ZKUI的zookeeper 3.4.10的docker镜像了
把该镜像上传到HARBOR库，稍后k8s编排时会用到。
- 通过Kubernetes 编排启动Zookeeper集群：
我们采取的方案是每个zkserver属于一个pod，每个pod绑定一个物理主机。
只需要完成yaml文件即可:zk-1.yaml:
---apiVersion: v1kind: Servicemetadata: name: zk-1-svc labels: app: zk-1-svcspec: ports: - port: 2888 name: server - port: 3888 name: leader-election - port: 2181 name: client - port: 9999 name: zkui selector: app: zk-1 type: NodePort---apiVersion: v1kind: ConfigMapmetadata: name: zk-1-cmdata: jvm.heap: "1G" tick: "2000" init: "10" sync: "5" client.cnxns: "60" snap.retain: "3" purge.interval: "0"---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: zk-1spec: replicas: 1 template: metadata: labels: app: zk-1 spec: nodeSelector: zk: zk-1 containers: - name: myzk imagePullPolicy: IfNotPresent image: registry.k8s.intra.knownsec.com/bigdata/zookeeper_3.4.10_zkui_2.0:0.0.1 resources: requests: memory: "2Gi" cpu: "500m" ports: - containerPort: 2181 name: client - containerPort: 2888 name: server - containerPort: 3888 name: leader-election - containerPort: 9999 name: zkui env: - name : ZK_HEAP_SIZE valueFrom: configMapKeyRef: name: zk-1-cm key: jvm.heap - name : ZK_TICK_TIME valueFrom: configMapKeyRef: name: zk-1-cm key: tick - name : ZK_INIT_LIMIT valueFrom: configMapKeyRef: name: zk-1-cm key: init - name : ZK_SYNC_LIMIT valueFrom: configMapKeyRef: name: zk-1-cm key: tick - name : ZK_MAX_CLIENT_CNXNS valueFrom: configMapKeyRef: name: zk-1-cm key: client.cnxns - name: ZK_SNAP_RETAIN_COUNT valueFrom: configMapKeyRef: name: zk-1-cm key: snap.retain - name: ZK_PURGE_INTERVAL valueFrom: configMapKeyRef: name: zk-1-cm key: purge.interval - name: ZK_CLIENT_PORT value: "2181" - name: ZK_SERVER_PORT value: "2888" - name: ZK_ELECTION_PORT value: "3888" - name: ZOO_MY_ID value: "1" - name: ZOO_SERVERS value: "server.1=0.0.0.0:2888:3888 server.2=zk-2-svc:2888:3888" - name: ZOO_UI_PORT value: "9999" - name: ZOO_UI_SERVER value: "zk-1-svc:2181,zk-2-svc:2181"# readinessProbe:# exec:# command:# - "zkok.sh"# initialDelaySeconds: 10# timeoutSeconds: 5# livenessProbe:# exec:# command:# - "zkok.sh"# initialDelaySeconds: 10# timeoutSeconds: 5 volumeMounts: - name: data mountPath: /data - name: datalog mountPath: /datalog volumes: - name: data hostPath: path: /data/k8s/zookeeper/data - name: datalog hostPath: path: /data/k8s/zookeeper/datalogzk-2.yaml:
---apiVersion: v1kind: Servicemetadata: name: zk-2-svc labels: app: zk-2-svcspec: ports: - port: 2888 name: server - port: 3888 name: leader-election - port: 2181 name: client - port: 9999 name: zkui selector: app: zk-2 type: NodePort---apiVersion: v1kind: ConfigMapmetadata: name: zk-2-cmdata: jvm.heap: "1G" tick: "2000" init: "10" sync: "5" client.cnxns: "60" snap.retain: "3" purge.interval: "0"---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: zk-2spec: replicas: 1 template: metadata: labels: app: zk-2 spec: nodeSelector: zk: zk-2 containers: - name: myzk imagePullPolicy: IfNotPresent image: registry.k8s.intra.knownsec.com/bigdata/zookeeper_3.4.10_zkui_2.0:0.0.1 resources: requests: memory: "2Gi" cpu: "500m" ports: - containerPort: 2181 name: client - containerPort: 2888 name: server - containerPort: 3888 name: leader-election - containerPort: 9999 name: zkui env: - name : ZK_HEAP_SIZE valueFrom: configMapKeyRef: name: zk-2-cm key: jvm.heap - name : ZK_TICK_TIME valueFrom: configMapKeyRef: name: zk-2-cm key: tick - name : ZK_INIT_LIMIT valueFrom: configMapKeyRef: name: zk-2-cm key: init - name : ZK_SYNC_LIMIT valueFrom: configMapKeyRef: name: zk-2-cm key: tick - name : ZK_MAX_CLIENT_CNXNS valueFrom: configMapKeyRef: name: zk-2-cm key: client.cnxns - name: ZK_SNAP_RETAIN_COUNT valueFrom: configMapKeyRef: name: zk-2-cm key: snap.retain - name: ZK_PURGE_INTERVAL valueFrom: configMapKeyRef: name: zk-2-cm key: purge.interval - name: ZK_CLIENT_PORT value: "2181" - name: ZK_SERVER_PORT value: "2888" - name: ZK_ELECTION_PORT value: "3888" - name: ZOO_MY_ID value: "2" - name: ZOO_SERVERS value: "server.1=zk-1-svc:2888:3888 server.2=0.0.0.0:2888:3888 " - name: ZOO_UI_PORT value: "9999" - name: ZOO_UI_SERVER value: "zk-1-svc:2181,zk-2-svc:2181"# readinessProbe:# exec:# command:# - "zkok.sh"# initialDelaySeconds: 10# timeoutSeconds: 5# livenessProbe:# exec:# command:# - "zkok.sh"# initialDelaySeconds: 10# timeoutSeconds: 5 volumeMounts: - name: data mountPath: /data - name: datalog mountPath: /datalog volumes: - name: data hostPath: path: /data/k8s/zookeeper/data - name: datalog hostPath: path: /data/k8s/zookeeper/datalog[yaml共享]http://pan.baidu.com/s/1jIyvGPK
执行：
➜ zookeeper kubectl create -f zk-1.yamlservice "zk-1-svc" createdconfigmap "zk-1-cm" createddeployment "zk-1" created➜ zookeeper kubectl create -f zk-2.yamlservice "zk-2-svc" createdconfigmap "zk-2-cm" createddeployment "zk-2" created
- 功能验证：
➜ zookeeper kubectl get po #看po情况NAME READY STATUS RESTARTS AGEzk-1-1238243890-phcm8 1/1 Running 0 1mzk-2-4022999611-x27vq 1/1 Running 0 1m➜ zookeeper kubectl get po -o wide #看绑定物理节点情况NAME READY STATUS RESTARTS AGE IP NODEzk-1-1238243890-phcm8 1/1 Running 0 1m 11.233.112.152 yuyan1zk-2-4022999611-x27vq 1/1 Running 0 1m 11.233.77.83 yuyan2➜ zookeeper kubectl get service #看端口映射情况NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGEkubernetes 11.233.0.1 <none> 443/TCP 4dzk-1-svc 11.233.30.168 <nodes> 2888:31292/TCP,3888:30513/TCP,2181:31570/TCP,9999:30476/TCP 1mzk-2-svc 11.233.27.68 <nodes> 2888:30906/TCP,3888:30842/TCP,2181:32473/TCP,9999:31490/TCP 1m➜ zookeeper kubectl exec zk-2-4022999611-x27vq -t -i -- bash #进入容器看服务情况bash-4.3# zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: /conf/zoo.cfgMode: leader #选举正常通过宿主机ip加映射后的端口访问zkui http://10.0.8.183:31490
提供对外访问正常参考
[Kargo部署K8s]http://oo3p.com/2017/04/17/kargo容器化部署kubernetes高可用集群/
[zookeeper docker]https://github.com/31z4/zookeeper-docker/tree/f12428ab7c6ea263ef037cf258129b83276c009c
[k8s pod]https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
[k8s编排zookeeper集群]https://github.com/kubernetes/contrib/tree/master/statefulsets/zookeeper
作者：俺是亮哥
链接：http://www.jianshu.com/p/315e18c2f3a2
來源：简书
著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。
阅读全文
0 0
- Kubernetes部署大数据组件系列一：一键部署Zookeeper集群
- Kubernetes部署大数据组件系列二：一键部署Hadoop集群
- 大数据开发部署系列--zookeeper
- 【云星数据---大数据部集群署系列004】：zookeeper分布式部署方案
- salt一键部署zookeeper
- Hadoop系列一：Hadoop集群分布式部署
- 大数据集群部署
- [Zookeeper系列一]Zookeeper应用介绍与安装部署
- 【Zookeeper系列一】Zookeeper应用介绍与安装部署
- kubernetes集群部署
- kubernetes-ubuntu集群部署
- 部署kubernetes集群
- Centos7部署Kubernetes集群
- Kubernetes集群部署
- Centos7部署Kubernetes集群
- Centos7部署Kubernetes集群
- 用Kubernetes部署企业大数据应用
- ZooKeeper部署和应用(一)
- Java Socket编程
- session
- ZOJ
- 通用程序设计
- 批量修改图片大小——基于Opencv库
- Kubernetes部署大数据组件系列一：一键部署Zookeeper集群
- BeautifulSoup 入门指南
- 字典：当索引不好用时
- linux 下安装 pip，linux下安装Mysql-Python
- mysql jeesite 大量数据 count 问题
- 关于JAVA程序中的菱形打印问题
- gpio子系统和pinctrl子系统（上）
- 《TCP/IP详解 卷1》 笔记： 概述
- leetcode31. Next Permutation
