Docker全新的容器集群项目——SwarmKit初步测试

题记

对于Docker容器集群来说，比较成熟的方案包括Swarm、Mesos、k8s和Google的Kubernetes，特别是后者得到了更多厂商的使用和推广，但是kubernetes相比较来说技术门槛较高，让很多用户望而却步，幸好，Docker在今年的6月7号开源发布了原生的集群管理工具SwarmKit，主要提供容器集群以及编排能力，那赶紧尝试一下，看看有什么好的功能。

SwarmKit框架

SwarmKit中有两种角色，Manager和Worker。Manager主要管理节点、调度任务。Worker主要通过Executor来执行任务，当前缺省的Executor为Docker Container Executor。包含了一下特点：

（1）内建分布式存储，不要额外的数据库
（2）支持Rolling update
（3）容器HA，支持Zero applicaton downtime
（4）通过TLS保证通讯安全

接下来就赶紧安装一下。

             “ SwarmKit 环境部署”

部署环境

VMWare Workstation 12 

Ubuntu 14.04

先制作一个通用的Ubuntu14.04虚拟机，然后安装Docker、安装Swarmkit包，然后进行虚拟机复制，生成三台集群环境，分别为一个Manager和两个Worker。

1、安装Ubuntu 14.04

2、安装Docker

curl -sSL https://get.docker.io | bash

3、由于SwarmKit采用Go语言开发，所以需要部署一个go环境，下载go包

下载地址：http://www.golangtc.com/download

我这里下载的是go1.6.2.linux-amd64.tar.gz

4、解压缩到/usr/local下面即可

然后添加环境变量 至 /etc/profile

export GOROOT=/usr/local/goexport PATH=/root/go/src/github.com/docker/swarmkit/bin:/usr/local/go/bin:$PATHexport GOPATH=/root/goexport SWARM_SOCKET=/tmp/controller/swarm.sock

注意：我目前的环境变量包含了所有的

环境变量生效之后，直接输入go即可看到命令提示。

5、安装git包

apt-get install git

6、输入如下命令下载swarmkit包

$ go get github.com/docker/swarmkit

注意：需要配置GOPATH环境变量，如上所述：/root/go，相关代码会下载到该目录下

7、进入/root/go/src/github.com/docker/swarmkit目录，make即可

root@controller:~/go/src/github.com/docker/swarmkit# pwd/root/go/src/github.com/docker/swarmkitroot@controller:~/go/src/github.com/docker/swarmkit# lsagent  bin          ca          cmd          CONTRIBUTING.md  Godeps    ioutils  log          Makefile  NOMENCLATURE.md  protobuf   vendorapi    BUILDING.md  circle.yml  codecov.yml  doc.go           identity  LICENSE  MAINTAINERS  manager   picker           README.md  version

8、在该目录的bin文件夹下可以看到生成的二进制文件

root@controller:~/go/src/github.com/docker/swarmkit/bin# lltotal 76092drwxr-xr-x  2 root root     4096 Jun 30 10:00 ./drwxr-xr-x 17 root root     4096 Jun 30 10:00 ../-rwxr-xr-x  1 root root 12929120 Jun 30 10:00 protoc-gen-gogoswarm*-rwxr-xr-x  1 root root 18044776 Jun 30 10:00 swarm-bench*-rwxr-xr-x  1 root root 19219800 Jun 30 10:00 swarmctl*-rwxr-xr-x  1 root root 27708144 Jun 30 10:00 swarmd*

swarmd是一个swarmkit daemon程序，用来运行manager和worker。

swarmctl是一个命令行工具，用来访问swarm manger

我们需要将这四个文件分别拷贝到不同manage和worker节点的/usr/bin里面，当然，也可以放在上述的环境变量里面。

9、建议在该机器下载一个测试镜像。

10、关闭虚拟机，复制。

              “ SwarmKit 具体使用 ”

目前我已经准备好了三台虚拟机，里面的环境都是一样的。

Manager：192.168.14.244/controller

Worker：192.168.14.80/worker1

Worker：192.168.14.223/worker2

确保manager和所有worker都可以实现ssh无密码访问

1、在manager节点执行启动命令swarmd -d /tmp/controller --listen-control-api /tmp/controller/swarm.sock --hostname controller

root@controller:~# swarmd -d /tmp/controller --listen-control-api /tmp/controller/swarm.sock --hostname controllerWarning: Specifying a valid address with --listen-remote-api may be necessary for other managers to reach this one.INFO[0000] Listening for connections                     addr=[::]:4242 proto=tcpINFO[0000] Listening for local connections               addr=/tmp/controller/swarm.sock proto=unixINFO[0000] 2dacc8ed9d604bff became follower at term 0INFO[0000] newRaft 2dacc8ed9d604bff [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]INFO[0000] 2dacc8ed9d604bff became follower at term 1INFO[0000] 2dacc8ed9d604bff is starting a new election at term 1INFO[0000] 2dacc8ed9d604bff became candidate at term 2INFO[0000] 2dacc8ed9d604bff received vote from 2dacc8ed9d604bff at term 2INFO[0000] 2dacc8ed9d604bff became leader at term 2INFO[0000] raft.node: 2dacc8ed9d604bff elected leader 2dacc8ed9d604bff at term 2INFO[0000] node is ready

注意：你可以使用nohup命令启动后台运行

然后分别在worker机器上运行加入集群的命令

swarmd -d /tmp/work1 --hostname work1 --join-addr 192.168.13.244:4242

root@worker1:~# swarmd -d /tmp/work1 --hostname work1 --join-addr 192.168.13.244:4242Warning: Specifying a valid address with --listen-remote-api may be necessary for other managers to reach this one.INFO[0000] Waiting for TLS certificate to be issued...INFO[0000] Downloaded new TLS credentials with role: swarm-worker.INFO[0000] node is ready

swarmd -d /tmp/work2 --hostname work2 --join-addr 192.168.13.244:4242

root@worker2:~# swarmd -d /tmp/work2 --hostname work2 --join-addr 192.168.13.244:4242Warning: Specifying a valid address with --listen-remote-api may be necessary for other managers to reach this one.INFO[0000] Waiting for TLS certificate to be issued...INFO[0000] Downloaded new TLS credentials with role: swarm-worker.INFO[0000] node is ready

这样三台机器都加入了统一的SwarmKit集群管理中了。接下来我们就可以通过swarmctl命令对docker集群进行管理了，这里面需要添加SWARM_SOCKET环境变量，参看上述。

2、查看目前的集群节点情况

root@controller:~# swarmctl node lsID                         Name        Membership  Status  Availability  Manager Status--                         ----        ----------  ------  ------------  --------------5a9hk2li71cx0fuzgi42wsrvc  work1       ACCEPTED    READY   ACTIVEa84vwflvs2ao4awtxor4ybaa6  work2       ACCEPTED    READY   ACTIVEazwpol92a8mrk1mssqw2ry7p7  controller  ACCEPTED    READY   ACTIVE        REACHABLE *

可以看到目前所有的状态都是正常状态

3、创建一个ubuntu服务

root@controller:~# swarmctl service create --name ubuntu --image ubuntu:14.041601akon3w1t1m7i9el4pn9gl

注意：如果集群各个节点包含镜像，启动比较快，如果不包含，各个节点如果联网，自行去docker hub里面下载镜像，这个可能会花费时间。这也是我建议为什么先下载一个镜像的原因。

查看目前的服务情况

root@controller:~# swarmctl service lsID                         Name    Image         Replicas--                         ----    -----         --------1601akon3w1t1m7i9el4pn9gl  ubuntu  ubuntu:14.04  0/1

通过inspect查看服务的详细情况。

root@controller:~# swarmctl service inspect ubuntuID                : 1601akon3w1t1m7i9el4pn9glName              : ubuntuReplicas          : 0/1Template Container  Image           : ubuntu:14.04Task ID                      Service    Slot    Image           Desired State    Last State                  Node-------                      -------    ----    -----           -------------    ----------                  ----9qflavdz3okfqtael5558o8ey    ubuntu     1       ubuntu:14.04    RUNNING          PREPARING 34 seconds ago    controller

Last State是PREPARING，说明controller还没有启动ubuntu容器，因为本地还没有镜像，需要从镜像仓库拉取。

4、更新服务

SwarmKit提供的swarmctl service  update功能可以对已有的服务的信息进行更新，例如镜像版本，实例参数（CPU、内存、端口号、网络、卷...）、副本数、标签、环境变量等进行更新

root@controller:~# swarmctl service update ubuntuError: no changes detectedUsage:  swarmctl service update <service ID> [flags]Flags:      --args value                  container args (default [])      --bind value                  define a bind mount (default [])      --command value               override entrypoint (default [])      --constraint value            Placement constraint (node.labels.key==value) (default [])      --cpu-limit string            CPU cores limit (e.g. 0.5)      --cpu-reservation string      number of CPU cores reserved (e.g. 0.5)      --env value                   container env (default [])      --image string                container image      --label value                 service label (key=value) (default [])      --memory-limit string         memory limit (e.g. 512m)      --memory-reservation string   amount of reserved memory (e.g. 512m)      --name string                 service name      --network string              network name      --ports value                 ports (default [])      --replicas uint               number of replicas for the service (only works in replicated service mode) (default 1)      --restart-condition string    condition to restart the task (any, failure, none) (default "any")      --restart-delay string        delay between task restarts (default "5s")      --restart-max-attempts uint   maximum number of restart attempts (0 = unlimited)      --restart-window string       time window to evaluate restart attempts (0 = unbound) (default "0s")      --update-delay string         delay between task updates (0s = none) (default "0s")      --update-parallelism uint     task update parallelism (0 = all at once)      --volume value                define a volume mount (default [])

5、节点管理

root@controller:~# swarmctl service inspect ubuntuID                : 1601akon3w1t1m7i9el4pn9glName              : ubuntuReplicas          : 0/3Template Container  Image           : ubuntu:14.04Task ID                      Service    Slot    Image           Desired State    Last State                  Node-------                      -------    ----    -----           -------------    ----------                  ----e4xaodj9v2sest4ix141wgu4t    ubuntu     1       ubuntu:14.04    ACCEPTED         ACCEPTED now                work20yfsmmpvq51clsywnfjn077va    ubuntu     2       ubuntu:14.04    RUNNING          PREPARING 17 minutes ago    work1706418wj5je0h7jtirvqmhlfi    ubuntu     3       ubuntu:14.04    ACCEPTED         ACCEPTED now                controller

我们可以通过swarmctl node drain work2命令将work2设置为不可用状态

（如果希望激活，使用swarmctl node activate work2命令即可）

同时我们可以查看到状态改变

root@controller:~# swarmctl node drain work2root@controller:~# swarmctl service inspect ubuntuID                : 1601akon3w1t1m7i9el4pn9glName              : ubuntuReplicas          : 0/3Template Container  Image           : ubuntu:14.04Task ID                      Service    Slot    Image           Desired State    Last State                  Node-------                      -------    ----    -----           -------------    ----------                  ----2znvxhamkbvj2eqf5q3xxin7d    ubuntu     1       ubuntu:14.04    RUNNING          PREPARING 6 seconds ago     controller0yfsmmpvq51clsywnfjn077va    ubuntu     2       ubuntu:14.04    RUNNING          PREPARING 19 minutes ago    work1890be0l7eapjep5v8k70b2pto    ubuntu     3       ubuntu:14.04    ACCEPTED         ACCEPTED 2 seconds ago      controller

我们可以看到

任务已经迁移到controller和work1上了。

后面可以看到controller的实例以ubuntu.3.xxxxx；work1的实例以ubuntu.2.xxxxx；work2的实例名以ubuntu.1.xxxxx，这是因为我执行了（swarmctl service update redis --replicas 3）副本操作。

我们看到在controller里面已经可以看到原来work2的实例了

root@controller:~# docker ps -aCONTAINER ID        IMAGE               COMMAND             CREATED              STATUS                          PORTS               NAMESa159185201f8        ubuntu:14.04        "/bin/bash"         5 seconds ago        Exited (0) 4 seconds ago                            ubuntu.1.58zd7f87vldcici33i272v20o0507647074e1        ubuntu:14.04        "/bin/bash"         8 seconds ago        Exited (0) 6 seconds ago                            ubuntu.3.dwbyeoyk9bgu07u2wz3utoput50a5aea24ed2        ubuntu:14.04        "/bin/bash"         17 seconds ago       Exited (0) 16 seconds ago                           ubuntu.1.9dwu5e3tb9cte2w2fnp7lyd1rf8244e6e4d55        ubuntu:14.04        "/bin/bash"         23 seconds ago       Exited (0) 22 seconds ago                           ubuntu.3.0dipyu39lyhvqtc68bq5tfld27019fbed41f1        ubuntu:14.04        "/bin/bash"         31 seconds ago       Exited (0) 29 seconds ago                           ubuntu.1.2kjo445p3l2fucgo82e06gy1ea63eac32c424        ubuntu:14.04        "/bin/bash"         33 seconds ago       Exited (0) 31 seconds ago                           ubuntu.3.anr48rmhqyh98v2xmwvafb60x928882228a57        ubuntu:14.04        "/bin/bash"         46 seconds ago       Exited (0) 45 seconds ago                           ubuntu.3.bymam97t2g6libynguczc6lm86ec7ad7a6bd2        ubuntu:14.04        "/bin/bash"         48 seconds ago       Exited (0) 47 seconds ago                           ubuntu.1.ct3fc130hki4keq6ax0uq0at698e9728e9000        ubuntu:14.04        "/bin/bash"         56 seconds ago       Exited (0) 55 seconds ago                           ubuntu.3.arjd0dtjefhofd5xnxw48lolibcf0708aae66        ubuntu:14.04        "/bin/bash"         59 seconds ago       Exited (0) 58 seconds ago                           ubuntu.1.augk3ferw6tvwe8qs6fk3wtt257db27be5bbd        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.3.6tkx22dhta5gtmkbpk83osj4k6fdf69e49e98        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.1.4qamwbaezfp3iigo0kkwzh5my3715774ddda2        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.3.5h2ohspdob81pueb74xmrr9q922071586b2a0        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.1.f4lmuk63l20kp165qe6uovo1n797aa652b3da        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.3.2pyygimy8ilaub70gltsa96d022143a2f7795        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.1.dvrjxce69qa382otiw225gmac1a26f50f87fa        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.3.byb6g2iuehmvgif7ss0nwnk0df43232d8a05a        ubuntu:14.04        "/bin/bash"         2 minutes ago        Exited (0) About a minute ago                       ubuntu.1.8zucjezgh65dv58jckgt0ycjj

后面我们将work2激活后，看到work2的实例又启用了

root@worker2:~# docker ps -aCONTAINER ID        IMAGE               COMMAND             CREATED              STATUS                          PORTS               NAMES9424a0b3dd72        ubuntu:14.04        "/bin/bash"         10 seconds ago       Exited (0) 9 seconds ago                            ubuntu.1.ezptv6gxoalsqdawl9uz2anwk6dd10817d0cf        ubuntu:14.04        "/bin/bash"         20 seconds ago       Exited (0) 19 seconds ago                           ubuntu.1.8lnrs64khjyqj69cnt03o20csa71f20118aba        ubuntu:14.04        "/bin/bash"         32 seconds ago       Exited (0) 31 seconds ago                           ubuntu.1.2i81636zz5vpkzjxs0emswmysef939614adcc        ubuntu:14.04        "/bin/bash"         42 seconds ago       Exited (0) 41 seconds ago                           ubuntu.1.0b66k4exrehqc1zatoalmvfkie06a3e77986f        ubuntu:14.04        "/bin/bash"         52 seconds ago       Exited (0) 52 seconds ago                           ubuntu.1.41gzjjru89aym5yxx56f1li77c9a5e2ecacfd        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.1.5hwusgfhb20fkzo1nloragkia0785dd33cdf1        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.1.71qca8ze2rpb94p3zv86y4u0e6fc4a1a657aa        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.1.0vujqtkoz7bl3rtui2kpexjhmb2c046e8515c        ubuntu:14.04        "/bin/bash"         About a minute ago   Exited (0) About a minute ago                       ubuntu.1.54elpfbr77jaq420mar5hn3vi

结语

目前SwarmKit由于刚开源，肯定有大量的问题存在，而且都还不能进入生产系统，但是作为直接集成到Docker Engine的优势比较明显，我个人在使用过程中还发现了不少问题。（问题可能是我理解不同）

例如：为什么我频繁刷新某个机器的docker实例，相关的Contrainer_ID并不一样，频繁变化，如果我全部启动后，contrainer_id固定下来了，但是类似上述的迁移还发生了变化。

另外，我理解的集群管理应该是我创建一个镜像的服务，应该根据内部的调度算法，创建某个容器实例在某个集群节点上，（上述看到的很多，是因为我更新了服务副本为3，执行了swarmctl service update redis --replicas 3），

但是为什么每个集群节点都包含9个容器实例，难道是根据我的虚拟机配置2vCPU+2vG，直接占满资源么？

后面还需要对该工具进一步研究，不过我还是比较看好该工具在容器集群的发展前景的。