CentOS7上Glusterfs的安装及使用（gluster/heketi)

1.glusterfs安装

安装并设置自启动

yum -y install centos-release-glusteryum -y install glusterfs-serversystemctl enable glusterdsystemctl start glusterd

配置每台机器hosts

vim /etc/hosts    10.132.47.78    lk-glusterfs-47-78     10.132.47.79    lk-glusterfs-47-79    10.132.47.80    lk-glusterfs-47-80

2.glusterfs使用

2.1 法一：通过gluster命令使用

2.1.1为存储池添加节点Node

gluster peer probe lk-glusterfs-47-78gluster peer probe lk-glusterfs-47-79gluster peer probe lk-glusterfs-47-80

2.1.2创建GlusterFS卷(以复本卷为例):

每台机器上以/gluster/gv0目录各创建一个brick:

mkdir -p /gluster/gv0

以每台机器的brick创建一个有3复本的逻辑卷gv0：

gluster volume create  gv0 replica 3 lk-glusterfs-47-78:/gluster/gv0 lk-glusterfs-47-79:/gluster/gv0 lk-glusterfs-47-80:/gluster/gv0 force  

启用volume:

gluster volume start gv0            

附：其他相关操作

从GlusterFS卷gv0移除某一brick：    gluster volume remove-brick gv0 replica 2 lk-glusterfs-47-79:/gluster/gv0 force删除GlusterFS卷gv0：    需要先stop卷：        gluster volume stop gv0    再删：        gluster volume delete gv0

2.1.3 gluster-client使用glusterfs卷

1.安装glusterfs client客户端:

yum -y install centos-release-glusteryum -y install glusterfs-client

2./etc/hosts中添加glusterfs-server信息：

vim /etc/hosts      10.132.47.78    lk-glusterfs-47-78    10.132.47.79    lk-glusterfs-47-79    10.132.47.80    lk-glusterfs-47-80

3.client挂载gv0卷到/mnt/glusterfs目录并使用：

mkdir /mnt/glusterfsmount -t glusterfs lk-glusterfs-47-79:/gv0 /mnt/glusterfs

2.2 法二：通过Heketi提供的restapi使用

2.2.1 Heketi安装(yum部署)

1.安装（装在某台节点上，如10.132.47.79）：

yum -y install heketi heketi-client

2.配置秘钥对：
Heketi使用SSH来配置GlusterFS的所有节点。创建SSH密钥对:

ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''chown heketi:heketi /etc/heketi/heketi_key*

制作完成后会在当前目录下生成heketi_key、heketi_key.pub，将公钥heketi_key.pub拷贝到所有glusterfs节点上/etc/heketi/keketi_key.pub（包括你登陆的第一个节点）,/etc/heketi/heketi.json 中 的 keyfile 指向 生成的 key（包含路径）

ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.132.47.78ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.132.47.79ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.132.47.80

备注，以上一句等效于以下三句：

scp /etc/heketi/heketi_key.pub root@10.132.47.78:/tmpssh root@10.132.47.78cat /tmp/id_rsa.pub >> /root/.ssh/authorized_keys

3.创建存储db的文件夹：

mkdir /dcos/heketichown -R heketi:heketi /dcos/heketi

4.配置 heketi.json
这里需要注意只是测试的话用mock 授权，standalone模式就 ssh 授权，k8s下就 kubernetes授权

vim /etc/heketi/heketi.json{  "_port_comment": "Heketi Server Port Number",  "port": "8088",  "_use_auth": "Enable JWT authorization. Please enable for deployment",  "use_auth": false,  "_jwt": "Private keys for access",  "jwt": {    "_admin": "Admin has access to all APIs",    "admin": {      "key": "123456"    },    "_user": "User only has access to /volumes endpoint",    "user": {      "key": "123456"    }  },  "_glusterfs_comment": "GlusterFS Configuration",  "glusterfs": {    "_executor_comment": [      "Execute plugin. Possible choices: mock, ssh",      "mock: This setting is used for testing and development.",      "      It will not send commands to any node.",      "ssh:  This setting will notify Heketi to ssh to the nodes.",      "      It will need the values in sshexec to be configured."    ],    "executor": "ssh",    "_sshexec_comment": "SSH username and private key file information",    "sshexec": {      "keyfile": "/etc/heketi/heketi_key",      "user": "root"    },    "_db_comment": "Database file name",    "db": "/dcos/heketi/heketi.db"  }}

systemctl enable heketi
systemctl restart heketi

5.测试heketi好用：

curl http://localhost:8088/hello

2.2.2 Heketi安装(容器化部署)

You will need to create a directory which has a directory containing configuraiton and any private key if necessary, and an empty directory used for storing the database. Directory and files must be read/write by user with id 1000 and if an ssh private key is used, it must also have a mod of 0600.

mkdir -p /dcos/heketi-docker/configmkdir -p /dcos/heketi-docker/dbcp /etc/heketi/heketi.json /dcos/heketi-docker/config/                      ---复用2.2.1的heketi.jsoncp /etc/heketi/heketi_key /dcos/heketi-docker/config/                       ---复用2.2.1的heketi_keychmod 600 /dcos/heketi-docker/config/heketi_keychown 1000:1000 -R /dcos/heketi-dockerTo run:# docker run --name=heketi -d -p 8089:8088 \             -v /dcos/heketi-docker/config:/etc/heketi \             -v /dcos/heketi-docker/db:/dcos/heketi \             -v /etc/hosts:/etc/hosts \             -v /etc/localtime:/etc/localtime \             heketi/heketi:4

更进一步，可以将heketi部署到k8s上：
chmod 600 /dcos/heketi-docker/config/heketi_key
chown 1000:1000 -R /dcos/heketi-docker
heketi.yaml:

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: heketi  namespace: kube-system  labels:    app: heketispec:  replicas: 1  selector:    matchLabels:      app: heketi  template:    metadata:      labels:        app: heketi    spec:      containers:        - name: heketi          image: heketi/heketi:4          imagePullPolicy: IfNotPresent          volumeMounts:            - name: config              mountPath: /etc/heketi            - name: db              mountPath: /dcos/heketi            - name: time              mountPath: /etc/localtime            - name: hosts              mountPath: /etc/hosts          ports:            - containerPort: 8088              name: heketi-api      volumes:        - name: config          hostPath:            path: /dcos/heketi-docker/config        - name: db          hostPath:            path: /dcos/heketi-docker/db        - name: time          hostPath:            path: /etc/localtime        - name: hosts          hostPath:            path: /etc/hosts      nodeSelector:        kubernetes.io/hostname: k8smaster01---apiVersion: v1kind: Servicemetadata:  labels:    app: heketi  name: heketi  namespace: kube-systemspec:  type: NodePort  ports:  - name: heketi    port: 8089    protocol: TCP    targetPort: 8088    nodePort: 30089  selector:    app: heketi

2.2.3 heketi-cli使用

Heketi集群初始化

1.创建cluster:

    heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true cluster create        {"id":"353fe5b92fa2a4d8671b6efdac8a5fd5","nodes":[],"volumes":[]}

2.将3个节点作为node添加到cluster:

    heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true node add --cluster="353fe5b92fa2a4d8671b6efdac8a5fd5" --management-host-name=10.132.47.78 --storage-host-name=10.132.47.78 --zone=1        {"zone":1,"hostnames":{"manage":["10.132.47.78"],"storage":["10.132.47.78"]},"cluster":"353fe5b92fa2a4d8671b6efdac8a5fd5","id":"a117cd328d609acc15e88dc0b6ab4889","devices":[]})    heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true node add --cluster="353fe5b92fa2a4d8671b6efdac8a5fd5" --management-host-name=10.132.47.79 --storage-host-name=10.132.47.79 --zone=1        {"zone":1,"hostnames":{"manage":["10.132.47.79"],"storage":["10.132.47.79"]},"cluster":"353fe5b92fa2a4d8671b6efdac8a5fd5","id":"5d3fbc8d0ac62d9c950d81df2f274dc6","devices":[]}    heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true node add --cluster="353fe5b92fa2a4d8671b6efdac8a5fd5" --management-host-name=10.132.47.80 --storage-host-name=10.132.47.80 --zone=1        {"zone":1,"hostnames":{"manage":["10.132.47.80"],"storage":["10.132.47.80"]},"cluster":"353fe5b92fa2a4d8671b6efdac8a5fd5","id":"a302a8367f18524ab5236e99f09072fe","devices":[]}

注意：
对接k8s的话，上边这个必须management-host-name要用ip地址，不可以用域名，否则从controller-manager中可以看到报错：

glusterfs: failed to create endpoint Endpoints "glusterfs-dynamic-gluster-pvc1" is invalid: [subsets[0].addresses[0].ip: Invalid value: "lk-glusterfs-47-80": must be a valid IP address, (e.g. 10.9.8.7), subsets[0].addresses[1].ip: Invalid value: "lk-glusterfs-47-79": must be a valid IP address, (e.g. 10.9.8.7), subsets[0].addresses[2].ip: Invalid value: "lk-glusterfs-47-78": must be a valid IP address, (e.g. 10.9.8.7)]

3.每台设备node上各添加一块裸硬盘/dev/sdb（没创建过任何分区），创建device：

    heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true device add --name="/dev/sdb" --node="a117cd328d609acc15e88dc0b6ab4889"    heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true device add --name="/dev/sdb" --node="5d3fbc8d0ac62d9c950d81df2f274dc6"    heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 --json=true device add --name="/dev/sdb" --node="a302a8367f18524ab5236e99f09072fe"

其实以上三步可以简化成这种方式：

1.创建topology.json文件：

vim /etc/heketi/topology.json{  "clusters": [    {      "nodes": [        {          "node": {            "hostnames": {              "manage": [                "lk-glusterfs-47-78"              ],              "storage": [                "10.132.47.78"              ]            },            "zone": 1          },          "devices": [            "/dev/sdb"          ]        },        {          "node": {            "hostnames": {              "manage": [                "lk-glusterfs-47-79"              ],              "storage": [                "10.132.47.79"              ]            },            "zone": 1          },          "devices": [            "/dev/sdb"          ]        },        {          "node": {            "hostnames": {              "manage": [                "lk-glusterfs-47-80"              ],              "storage": [                "10.132.47.80"              ]            },            "zone": 1          },          "devices": [            "/dev/sdb"          ]        }      ]    }  ]}

该文件格式比较简单，基本上是告诉heketi要创建一个3节点的集群，其中每个节点包含的配置有FQDN，IP地址以及至少一个将用作GlusterFS块的备用块设备。

2.将该文件发送给heketi创建

以systemd起heketi:    heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 topology load --json=/etc/heketi/topology.json以容器起heketi:    docker exec 容器ID heketi-cli --server http://10.142.21.23:30088 --user admin --secret 123456 topology load --json=/etc/heketi/topology.json

结果：
创建成功后，heketi会在每个gluster节点上创建一个逻辑卷组(vg_2d8771e16bfe0b267fe2b7133584af43)，通过vgscan或vgdisplay可以看到

创建volume：

以systemd起heketi:    heketi-cli --server http://10.132.47.79:8088 --user admin --secret 123456 volume create --size=100 --replica=3 --clusters=d691b29a06374c4da7e94bba71d027bf以容器起heketi:    docker exec 容器ID heketi-cli --server http://10.142.21.23:30088 --user admin --secret 123456  volume create --size=100 --replica=3 --clusters=d691b29a06374c4da7e94bba71d027bf

结果：
结果是在相应节点的逻辑卷组（vg_2d8771e16bfe0b267fe2b7133584af43）下会创建逻辑卷(/var/lib/heketi/mounts/vg_7157a2d1d7899269823997ad62e6debd/brick_1185ef33c9719d9063ccc2ccd0df96e7/brick)作为glusterfs的brick

gluster volume infoVolume Name: vol_50b5237866378d655af326a74fc7d68cType: Distributed-ReplicateVolume ID: b82dc1b3-b59c-48cd-ace1-155a0c05c42fStatus: StartedSnapshot Count: 0Number of Bricks: 2 x 3 = 6Transport-type: tcpBricks:Brick1: 10.142.21.22:/var/lib/heketi/mounts/vg_7157a2d1d7899269823997ad62e6debd/brick_1185ef33c9719d9063ccc2ccd0df96e7/brickBrick2: 10.142.21.27:/var/lib/heketi/mounts/vg_a850891bc0f47849bfdbdb115ee19656/brick_75894b35ae36dd5253bd79df1539b970/brickBrick3: 10.142.21.21:/var/lib/heketi/mounts/vg_78a29014ac0df3d31b1a357e096e8917/brick_3423914cb4d9873c670f5fab0ffe44e5/brickBrick4: 10.142.21.25:/var/lib/heketi/mounts/vg_25cc714589cec30e3550297cced4ce44/brick_6ca46dd4a13f7ac0b598d83fbf43c2fc/brickBrick5: 10.142.21.26:/var/lib/heketi/mounts/vg_12e5dbb7b9372e0d72b8ec2166b68048/brick_87e969be58d93ff7deba65587d2e5637/brickBrick6: 10.142.21.24:/var/lib/heketi/mounts/vg_54b5dc50bfe00644a39e289d9abf7c99/brick_5a21762ac6c5ab35472648455310af56/brickOptions Reconfigured:transport.address-family: inetnfs.disable: onheketi-cli --server http://10.142.21.21:8088 --user admin --secret 123456 topology info

参考：https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart
https://jimmysong.io/kubernetes-handbook/practice/storage-for-containers-using-glusterfs-with-openshift.html