zookeeper docker集群配置

本篇讲一下zookeeper在docker中集群搭建方式。

先来看看zookeeper的单个Docker配置吧。

共包含两个文件，一个Dockerfile，一个sh脚本

Dockerfile如下：

FROM openjdk:8-jre-alpine# Install required packagesRUN apk add --no-cache \    bash \    su-execENV ZOO_USER=zookeeper \    ZOO_CONF_DIR=/conf \    ZOO_DATA_DIR=/data \    ZOO_DATA_LOG_DIR=/datalog \    ZOO_PORT=2181 \    ZOO_TICK_TIME=2000 \    ZOO_INIT_LIMIT=5 \    ZOO_SYNC_LIMIT=2 \    ZOO_MAX_CLIENT_CNXNS=60# Add a user and make dirsRUN set -ex; \    adduser -D "$ZOO_USER"; \    mkdir -p "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR"; \    chown "$ZOO_USER:$ZOO_USER" "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR"ARG GPG_KEY=D0BC8D8A4E90A40AFDFC43B3E22A746A68E327C1ARG DISTRO_NAME=zookeeper-3.4.11# Download Apache Zookeeper, verify its PGP signature, untar and clean upRUN set -ex; \    apk add --no-cache --virtual .build-deps \        ca-certificates \        gnupg \        libressl; \    wget -q "https://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz"; \    wget -q "https://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz.asc"; \    export GNUPGHOME="$(mktemp -d)"; \    gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" || \    gpg --keyserver pgp.mit.edu --recv-keys "$GPG_KEY" || \    gpg --keyserver keyserver.pgp.com --recv-keys "$GPG_KEY"; \    gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz"; \    tar -xzf "$DISTRO_NAME.tar.gz"; \    mv "$DISTRO_NAME/conf/"* "$ZOO_CONF_DIR"; \    rm -rf "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc"; \    apk del .build-depsWORKDIR $DISTRO_NAMEVOLUME ["$ZOO_DATA_DIR", "$ZOO_DATA_LOG_DIR"]EXPOSE $ZOO_PORT 2888 3888ENV PATH=$PATH:/$DISTRO_NAME/bin \    ZOOCFGDIR=$ZOO_CONF_DIRCOPY docker-entrypoint.sh /RUN chmod 777 /docker-entrypoint.shENTRYPOINT ["/docker-entrypoint.sh"]CMD ["zkServer.sh", "start-foreground"]

docker-entrypoint.sh如下

#!/bin/bashset -e# Allow the container to be started with `--user`if [[ "$1" = 'zkServer.sh' && "$(id -u)" = '0' ]]; then    chown -R "$ZOO_USER" "$ZOO_DATA_DIR" "$ZOO_DATA_LOG_DIR"    exec su-exec "$ZOO_USER" "$0" "$@"fi# Generate the config only if it doesn't existif [[ ! -f "$ZOO_CONF_DIR/zoo.cfg" ]]; then    CONFIG="$ZOO_CONF_DIR/zoo.cfg"    echo "clientPort=$ZOO_PORT" >> "$CONFIG"    echo "dataDir=$ZOO_DATA_DIR" >> "$CONFIG"    echo "dataLogDir=$ZOO_DATA_LOG_DIR" >> "$CONFIG"    echo "tickTime=$ZOO_TICK_TIME" >> "$CONFIG"    echo "initLimit=$ZOO_INIT_LIMIT" >> "$CONFIG"    echo "syncLimit=$ZOO_SYNC_LIMIT" >> "$CONFIG"    echo "maxClientCnxns=$ZOO_MAX_CLIENT_CNXNS" >> "$CONFIG"    for server in $ZOO_SERVERS; do        echo "$server" >> "$CONFIG"    donefi# Write myid only if it doesn't existif [[ ! -f "$ZOO_DATA_DIR/myid" ]]; then    echo "${ZOO_MY_ID:-1}" > "$ZOO_DATA_DIR/myid"fiexec "$@"

有了这两个文件就可以打镜像了，譬如使用Daocloud即可，将那两个文件放GitHub里，用Daocloud创建一个项目指向该GitHub项目，然后就能打包镜像了。

从shell脚本可以看到，有几个项是可以定制的，主要就是zoo.cfg里面的东西，包括ZOO_MY_ID,ZOO_SERVERS,ZOO_TICK_TIME等等各个属性，都可以在docker启动时的环境变量里指定。

比较有用的就是ZOO_SERVERS和ZOO_MY_ID，是在搭建集群时需要指定的自己的id和其他zookeeper服务器的地址。

如果是单体zookeeper，直接用上面构建好的镜像启动就OK了。

搭建集群，如果是用docker-compose的方式，参考如下

version: '3.1'services:    zoo1:        image: xxxx/zookeeper        restart: always        hostname: zoo1        ports:            - 2181:2181        environment:            ZOO_MY_ID: 1            ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888    zoo2:        image: xxxx/zookeeper        restart: always        hostname: zoo2        ports:            - 2182:2181        environment:            ZOO_MY_ID: 2            ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zoo3:2888:3888    zoo3:        image: xxxx/zookeeper        restart: always        hostname: zoo3        ports:            - 2183:2181        environment:            ZOO_MY_ID: 3            ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=0.0.0.0:2888:3888

将image地址换成你自己的镜像地址即可。

我是将集群用rancher搭建的，看一下rancher的配置：

zookeeper1：

zookeeper2：

zookeeper3的也类似，就是添加另外两个服务的服务链接（即是docker link），然后设置环境变量即可。

3个镜像都启动后，zookeeper集群就算搭建完毕了。

可以分别进入容器1，执行创建node，然后在容器2和容器3分别查看该node，看集群成功的效果。

下面我们再使用zkui这个带界面的zookeeper可视化程序，也将它部署到docker里，来查看zookeeper集群的状态。

zkui的jar包可以自行去打包，或者网上去找个下载。最终用来打包镜像的目录如下：

Dockerfile如下：

FROM java:8WORKDIR /var/appADD zkui-*.jar /var/app/zkui.jarADD config.cfg /var/app/config.cfgADD bootstrap.sh /var/app/bootstrap.shRUN chmod 777 /var/app/bootstrap.shENTRYPOINT ["/var/app/bootstrap.sh"]EXPOSE 9090

bootstrap.sh

#!/bin/shZK_SERVER=${ZK_SERVER:-"localhost:2181"}USER_SET=${USER_SET:-"{\"users\": [{ \"username\":\"admin\" , \"password\":\"manager\",\"role\": \"ADMIN\" \},{ \"username\":\"appconfig\" , \"password\":\"appconfig\",\"role\": \"USER\" \}]\}"}LOGIN_MESSAGE=${LOGIN_MESSAGE:-"Please login using admin/manager or appconfig/appconfig."}sed -i "s/^zkServer=.*$/zkServer=$ZK_SERVER/" /var/app/config.cfgsed -i "s/^userSet = .*$/userSet = $USER_SET/" /var/app/config.cfgsed -i "s/^loginMessage=.*$/loginMessage=$LOGIN_MESSAGE/" /var/app/config.cfgecho "Starting zkui with server $ZK_SERVER"exec java -jar /var/app/zkui.jar

config.cfg

#Server PortserverPort=9090#Comma seperated list of all the zookeeper serverszkServer=localhost:2181,localhost:2181#Http path of the repository. Ignore if you dont intent to upload files from repository.scmRepo=http://myserver.com/@rev1=#Path appended to the repo url. Ignore if you dont intent to upload files from repository.scmRepoPath=//appconfig.txt#if set to true then userSet is used for authentication, else ldap authentication is used.ldapAuth=falseldapDomain=mycompany,mydomain#ldap authentication url. Ignore if using file based authentication.ldapUrl=ldap://<ldap_host>:<ldap_port>/dc=mycom,dc=com#Specific roles for ldap authenticated users. Ignore if using file based authentication.ldapRoleSet={"users": [{ "username":"domain\\user1" , "role": "ADMIN" }]}userSet = {"users": [{ "username":"admin" , "password":"manager","role": "ADMIN" },{ "username":"appconfig" , "password":"appconfig","role": "USER" }]}#Set to prod in production and dev in local. Setting to dev will clear history each time.env=prodjdbcClass=org.h2.DriverjdbcUrl=jdbc:h2:zkuijdbcUser=rootjdbcPwd=manager#If you want to use mysql db to store history then comment the h2 db section.#jdbcClass=com.mysql.jdbc.Driver#jdbcUrl=jdbc:mysql://localhost:3306/zkui#jdbcUser=root#jdbcPwd=managerloginMessage=Please login using admin/manager or appconfig/appconfig.#session timeout 5 mins/300 secs.sessionTimeout=300#Default 5 seconds to keep short lived zk sessions. If you have large data then the read will take more than 30 seconds so increase this accordingly. #A bigger zkSessionTimeout means the connection will be held longer and resource consumption will be high.zkSessionTimeout=5#Block PWD exposure over rest call.blockPwdOverRest=false#ignore rest of the props below if https=false.https=falsekeystoreFile=/home/user/keystore.jkskeystorePwd=passwordkeystoreManagerPwd=password# The default ACL to use for all creation of nodes. If left blank, then all nodes will be universally accessible# Permissions are based on single character flags: c (Create), r (read), w (write), d (delete), a (admin), * (all)# For example defaultAcl={"acls": [{"scheme":"ip", "id":"192.168.1.192", "perms":"*"}, {"scheme":"ip", id":"192.168.1.0/24", "perms":"r"}]defaultAcl=# Set X-Forwarded-For to true if zkui is behind a proxyX-Forwarded-For=false

可以看到，需要指定的就是zookeeper server的地址就行了，打成镜像后，部署到rancher中：

我这里只指了一个server，可以指定多个，譬如你把3个zookeeper server都放进去，用逗号分隔。

启动后访问界面，用 admin账号、manager密码登录

可以用命令行在随便一个zookeeper里创建node，然后在zkui界面查看。