osd 故障测试
来源：互联网 发布：iphone下载电影软件 编辑：程序博客网 时间：2024/05/04 19:30
1. 创建的 volumes 是否可以导出数据
查询当前池
[root@tt-ceph-048146 ~]# ceph osd lspools0 rbd,1 volumes,
查询 volumes 池中的卷
[root@tt-ceph-048144 ceph]# rbd -p volumes lsvolume-be34745e-165f-4fc6-86c4-6248173abc00
导出卷
[root@tt-ceph-048144 ceph]# rbd -p volumes export be34745e-165f-4fc6-86c4-6248173abc00 /tmp/volume-test2015-08-04 17:07:46.071512 7fcec6ada880 -1 librbd::ImageCtx: error finding header: (2) No such file or directoryrbd: error opening image be34745e-165f-4fc6-86c4-6248173abc00: (2) No such file or directory[root@tt-ceph-048144 ceph]# rbd -p volumes export volume-be34745e-165f-4fc6-86c4-6248173abc00 /tmp/volume-test
导出的卷只能够用于 ceph 导出, 而不可以令 linux 直接执行读写
[root@tt-ceph-048144 ceph]# dumpe2fs /tmp/volume-testdumpe2fs 1.42.9 (28-Dec-2013)dumpe2fs: Bad magic number in super-block 当尝试打开 /tmp/volume-test 时找不到有效的文件系统超级块.
2. vm 进行数据写入时候, 是否会对 mon 节点保持长连接, 是否会对 ceph osd 节点保持长连接
经测试, 当 vm 云盘后, 对应物理机即会产生一常连接到 mon 节点, 只有在 vm 产生数据写时候, 才会与 osd 节点产生对应的 TCP 连接MON 节点的常连接并不会全部都集中到其中的一台 MON 节点中, 常连接会比较平均地分布到各个 mon 节点中参考生产环境上的 osd 连接
第一个 mon 节点, 有 13 个常连接[root@sh-ceph-128212 ~]# for ip in 10.198.128.212; do ssh $ip "netstat -nt | grep 6789" | awk '{print $5}' | grep -E -v '10.198.128.212|10.198.128.213|10.198.128.214|10.198.128.215|10.198.128.216|10.198.128.217|10.198.128.218|10.198.128.219' ; done10.198.128.30:4755710.198.128.35:5377110.198.128.37:5848510.198.128.32:3384910.198.128.38:4627210.198.128.32:3381910.198.128.33:4036310.198.128.31:4511010.198.128.32:5607910.198.128.30:5918710.198.128.32:3383610.198.128.30:3413210.198.128.38:46288 第二个 mon 节点, 有 7 个常连接[root@sh-ceph-128212 ~]# for ip in 10.198.128.214; do ssh $ip "netstat -nt | grep 6789" | awk '{print $5}' | grep -E -v '10.198.128.212|10.198.128.213|10.198.128.214|10.198.128.215|10.198.128.216|10.198.128.217|10.198.128.218|10.198.128.219' ; done10.198.129.72:4489610.198.128.38:4776410.198.128.33:5110110.198.128.38:5620610.198.128.32:4492910.198.128.31:3797610.198.128.37:56534 第三个 mon 节点, 有 12 个常连接[root@sh-ceph-128212 ~]# for ip in 10.198.128.216; do ssh $ip "netstat -nt | grep 6789" | awk '{print $5}' | grep -E -v '10.198.128.212|10.198.128.213|10.198.128.214|10.198.128.215|10.198.128.216|10.198.128.217|10.198.128.218|10.198.128.219' ; done10.198.128.31:3759010.198.128.32:4900910.198.128.36:5263210.198.128.36:5261510.198.128.30:5446410.198.128.32:4578310.198.128.36:4673310.198.128.35:3486610.198.128.36:4679310.198.128.37:5442110.198.128.33:5213510.198.128.36:467133. 当 vm 进行数据写入时候, 断开一个 mon 节点, 是否会对数据写入造成影响
当前测试 compute 192.168.209.106 mon 192.168.48.144, 192.168.48.146, 192.168.48.148
1. 当前只有 192.168.48.144 节点中出现与客户的常连接2. 让 compute 节点中的 vm 进行大数据复制 (1gb 左右数据复制), 产生了磁盘 IO3. 这个时候利用 kill 命令删除 192.168.48.144 的 mon 进程, 并利用 ps 命令进行验证4. 当 mon 进程被杀死后, 192.168.209.106 与 192.168.48.148 创建了新的 mon 常连接5. 数据复制没有被影响
mon 进程不会自动恢复, 需要手动对 mon 进行启动
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148 osdmap e245: 30 osds: 30 up, 30 in pgmap v1015: 10304 pgs, 2 pools, 4660 MB data, 1183 objects 46119 MB used, 854 GB / 899 GB avail 10304 active+clean重启启动了 mon 服务后, 恢复正常
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e246: 30 osds: 30 up, 30 in pgmap v1016: 10304 pgs, 2 pools, 4660 MB data, 1183 objects 46119 MB used, 854 GB / 899 GB avail 10304 active+clean4. 当 vm 进行数据写入时候, 断开一个 osd 节点, 是否会对数据写入造成影响
1. 在 vm 写入数据的过程中, 关闭一台 osd 节点2. 在数据完成后, 验证数据完成性3. 重启启动 osd 节点, 发现 ceph 集群会自动恢复4. ceph 集群完整性没有被破坏
在进行数据复制过程中, 把其中一台的 osd 进程全部 kill 掉
[root@tt-ceph-048144 ceph]# ceph osd tree2015-08-04 14:57:32.063320 7fccf41a9700 0 -- :/1007979 >> 192.168.48.144:6789/0 pipe(0x7fccf00298e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fccf0029b70).fault# id weight type name up/down reweight-1 30 root default-2 5 host tt-ceph-0481440 1 osd.0 up 11 1 osd.1 up 12 1 osd.2 up 13 1 osd.3 up 14 1 osd.4 up 1-3 5 host tt-ceph-0481455 1 osd.5 up 16 1 osd.6 up 17 1 osd.7 up 18 1 osd.8 up 19 1 osd.9 up 1-4 5 host tt-ceph-04814610 1 osd.10 up 111 1 osd.11 up 112 1 osd.12 up 113 1 osd.13 up 114 1 osd.14 up 1-5 5 host tt-ceph-04814715 1 osd.15 up 116 1 osd.16 up 117 1 osd.17 up 118 1 osd.18 up 119 1 osd.19 up 1-6 5 host tt-ceph-04814820 1 osd.20 up 121 1 osd.21 up 122 1 osd.22 up 123 1 osd.23 up 124 1 osd.24 up 1-7 5 host tt-ceph-04814925 1 osd.25 down 126 1 osd.26 down 127 1 osd.27 down 128 1 osd.28 down 129 1 osd.29 down 1
检查 ceph 集群, 发现发生故障
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 5176 pgs degraded; 5171 pgs stuck unclean; 5176 pgs undersized; recovery 471/2598 objects degraded (18.129%); 5/30 in osds are down; 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148 monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148 osdmap e239: 30 osds: 25 up, 30 in pgmap v924: 10304 pgs, 2 pools, 3392 MB data, 866 objects 42279 MB used, 858 GB / 899 GB avail 471/2598 objects degraded (18.129%) 5128 active+clean 5176 active+undersized+degraded重新启动 osd, 发现有数据恢复现象
[root@tt-ceph-048144 ceph]# ceph -s2015-08-04 14:58:26.309245 7f7eb818b700 0 -- :/1008324 >> 192.168.48.144:6789/0 pipe(0x7f7eb40298e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f7eb4029b70).fault cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 3089 pgs degraded; 3085 pgs stuck unclean; 3089 pgs undersized; recovery 325/2835 objects degraded (11.464%); 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148 monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148 osdmap e245: 30 osds: 30 up, 30 in pgmap v941: 10304 pgs, 2 pools, 3708 MB data, 945 objects 43958 MB used, 856 GB / 899 GB avail 325/2835 objects degraded (11.464%) 7215 active+clean 3089 active+undersized+degradedrecovery io 12881 kB/s, 3 objects/s client io 3969 kB/s wr, 2 op/s恢复后, 发现, 数据正常了
[root@tt-ceph-048144 ceph]# ceph -s2015-08-04 14:58:44.340787 7fd72bd78700 0 -- :/1008446 >> 192.168.48.144:6789/0 pipe(0x7fd7300298e0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd730029b70).fault cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148 monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148 osdmap e245: 30 osds: 30 up, 30 in pgmap v952: 10304 pgs, 2 pools, 3925 MB data, 999 objects 45058 MB used, 855 GB / 899 GB avail 10304 active+clean client io 13701 kB/s rd, 20372 kB/s wr, 119 op/s重启一下故障的 mon 服务
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 1 mons down, quorum 1,2 tt-ceph-048146,tt-ceph-048148 monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 14, quorum 1,2 tt-ceph-048146,tt-ceph-048148 osdmap e245: 30 osds: 30 up, 30 in pgmap v1015: 10304 pgs, 2 pools, 4660 MB data, 1183 objects 46119 MB used, 854 GB / 899 GB avail 10304 active+clean第二次测试
数据复制正常
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e248: 30 osds: 30 up, 30 in pgmap v1052: 10304 pgs, 2 pools, 4680 MB data, 1188 objects 46095 MB used, 854 GB / 899 GB avail 10304 active+clean client io 5402 kB/s rd, 3031 kB/s wr, 46 op/s对其中一台电脑中所有的 osd 执行 kill 操作
查询发现, osd 出现了故障, GP 也出现非正常现象
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 5176 pgs degraded; 5155 pgs stuck unclean; 5176 pgs undersized; recovery 930/4533 objects degraded (20.516%); 5/30 in osds are down monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e250: 30 osds: 25 up, 30 in pgmap v1098: 10304 pgs, 2 pools, 5966 MB data, 1511 objects 52202 MB used, 848 GB / 899 GB avail 930/4533 objects degraded (20.516%) 5128 active+clean 5176 active+undersized+degraded client io 16592 kB/s rd, 4918 B/s wr, 134 op/s在数据复制完成后, 重启启动所有 osd
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 147 pgs degraded; 234 pgs peering; 12 pgs recovering; 29 pgs recovery_wait; 73 pgs stuck unclean; recovery 467/4620 objects degraded (10.108%) monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e257: 30 osds: 28 up, 28 in pgmap v1146: 10304 pgs, 2 pools, 6080 MB data, 1540 objects 51945 MB used, 758 GB / 809 GB avail 467/4620 objects degraded (10.108%) 1 inactive 234 peering 9922 active+clean 106 active+degraded 29 active+recovery_wait+degraded 12 active+recovering+degraded再次查询, 数据自动恢复中
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 174 pgs degraded; 19 pgs recovering; 76 pgs recovery_wait; 93 pgs stuck unclean; recovery 586/4620 objects degraded (12.684%) monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e261: 30 osds: 30 up, 30 in pgmap v1156: 10304 pgs, 2 pools, 6080 MB data, 1540 objects 53205 MB used, 847 GB / 899 GB avail 586/4620 objects degraded (12.684%) 79 active+degraded 10130 active+clean 76 active+recovery_wait+degraded 19 active+recovering+degradedrecovery io 379 MB/s, 97 objects/s client io 9401 kB/s wr, 4 op/s再查询
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e261: 30 osds: 30 up, 30 in pgmap v1162: 10304 pgs, 2 pools, 6080 MB data, 1540 objects 53553 MB used, 847 GB / 899 GB avail 10304 active+cleanrecovery io 52337 kB/s, 12 objects/s最终, 恢复完成
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 16, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e261: 30 osds: 30 up, 30 in pgmap v1170: 10304 pgs, 2 pools, 6080 MB data, 1540 objects 51335 MB used, 849 GB / 899 GB avail 10304 active+clean5. 新增 osd 节点进行扩容
当前集群磁盘状态 ( tt-ceph-048144 ~ tt-ceph-048149)
[root@tt-ceph-048144 ceph]# ceph dfGLOBAL: SIZE AVAIL RAW USED %RAW USED 899G 724G 175G 19.50POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 235G 0 volumes 1 48041M 5.22 235G 12166
新加入节点, 执行下面操作
1. 主机名同步, 时间同步, 安装 ceph 软件2. 执行磁盘格式化, 创建 osd , 挂载对应磁盘
参考格式化脚本
#!/bin/bashLANG=en_USdisk=`fdisk -l | grep ^Disk | grep sectors | grep sd | grep -v sda | awk -F[:\ ] '{print $2}' | sort`yum install -y hdparmfor partition in $diskdo dd if=/dev/zero of=$partition bs=1M count=100 parted -s $partition mklabel msdos parted $partition mkpart primary xfs 1 100% hdparm -z "$partition"1 mkfs.xfs -f -i size=512 "$partition"1done参考创建 osd 脚本
#!/bin/bashLANG=en_USnum=30diskpart=`fdisk -l | grep Linux | grep -v sda | awk '{print $1}' | sort`for partition in $diskpartdo ceph osd create mkdir /var/lib/ceph/osd/ceph-$num echo $partition /var/lib/ceph/osd/ceph-$num xfs defaults 0 0 >> /etc/fstab let num++donemount -a参考磁盘挂载情况
[root@tt-ceph-048150 tmp]# mount | grep osd/dev/sdb1 on /var/lib/ceph/osd/ceph-30 type xfs (rw,relatime,attr2,inode64,noquota)/dev/sdc1 on /var/lib/ceph/osd/ceph-31 type xfs (rw,relatime,attr2,inode64,noquota)/dev/sdd1 on /var/lib/ceph/osd/ceph-32 type xfs (rw,relatime,attr2,inode64,noquota)/dev/sde1 on /var/lib/ceph/osd/ceph-33 type xfs (rw,relatime,attr2,inode64,noquota)/dev/sdf1 on /var/lib/ceph/osd/ceph-34 type xfs (rw,relatime,attr2,inode64,noquota)/dev/sdg1 on /var/lib/ceph/osd/ceph-35 type xfs (rw,relatime,attr2,inode64,noquota)
参考 osd tree
[root@tt-ceph-048144 ceph]# ceph osd tree# id weight type name up/down reweight-1 30 root default-2 5 host tt-ceph-0481440 1 osd.0 up 11 1 osd.1 up 12 1 osd.2 up 13 1 osd.3 up 14 1 osd.4 up 1-3 5 host tt-ceph-0481455 1 osd.5 up 16 1 osd.6 up 17 1 osd.7 up 18 1 osd.8 up 19 1 osd.9 up 1-4 5 host tt-ceph-04814610 1 osd.10 up 111 1 osd.11 up 112 1 osd.12 up 113 1 osd.13 up 114 1 osd.14 up 1-5 5 host tt-ceph-04814715 1 osd.15 up 116 1 osd.16 up 117 1 osd.17 up 118 1 osd.18 up 119 1 osd.19 up 1-6 5 host tt-ceph-04814820 1 osd.20 up 121 1 osd.21 up 122 1 osd.22 up 123 1 osd.23 up 124 1 osd.24 up 1-7 5 host tt-ceph-04814925 1 osd.25 up 126 1 osd.26 up 127 1 osd.27 up 128 1 osd.28 up 129 1 osd.29 up 130 0 osd.30 down 031 0 osd.31 down 032 0 osd.32 down 033 0 osd.33 down 034 0 osd.34 down 035 0 osd.35 down 0
3 初始化 osd
参考 初始化脚本
#!/bin/bashLANG=en_USnum=30diskpart=`fdisk -l | grep Linux | grep -v sda | awk '{print $1}' | sort`for partition in $diskpartdo ceph-osd -i $num --mkfs --mkkey --osd-uuid 60391f4d-734f-425d-89b2-1fcb26c876c8 let num++done4 授权 osd
参考授权脚本
#!/bin/bashLANG=en_USnum=30diskpart=`fdisk -l | grep Linux | grep -v sda | awk '{print $1}' | sort`for partition in $diskpartdo ceph auth add osd.$num osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-$num/keyring let num++done5. 修改 crush map
ceph osd crush add-bucket tt-ceph-048150 hostceph osd crush move tt-ceph-048150 root=defaultfor num in `seq 30 35`doceph osd crush add osd.$num 1.0 root=default host=tt-ceph-048150done
参考修改后的 osd tree
[root@tt-ceph-048150 tmp]# ceph osd tree# id weight type name up/down reweight-1 36 root default-2 5 host tt-ceph-0481440 1 osd.0 up 11 1 osd.1 up 12 1 osd.2 up 13 1 osd.3 up 14 1 osd.4 up 1-3 5 host tt-ceph-0481455 1 osd.5 up 16 1 osd.6 up 17 1 osd.7 up 18 1 osd.8 up 19 1 osd.9 up 1-4 5 host tt-ceph-04814610 1 osd.10 up 111 1 osd.11 up 112 1 osd.12 up 113 1 osd.13 up 114 1 osd.14 up 1-5 5 host tt-ceph-04814715 1 osd.15 up 116 1 osd.16 up 117 1 osd.17 up 118 1 osd.18 up 119 1 osd.19 up 1-6 5 host tt-ceph-04814820 1 osd.20 up 121 1 osd.21 up 122 1 osd.22 up 123 1 osd.23 up 124 1 osd.24 up 1-7 5 host tt-ceph-04814925 1 osd.25 up 126 1 osd.26 up 127 1 osd.27 up 128 1 osd.28 up 129 1 osd.29 up 1-8 6 host tt-ceph-04815030 1 osd.30 down 031 1 osd.31 down 032 1 osd.32 down 033 1 osd.33 down 034 1 osd.34 down 035 1 osd.35 down 0
6. 启动 osd
for num in `seq 30 35`dotouch /var/lib/ceph/osd/ceph-$num/sysvinit/etc/init.d/ceph -a start osd.$numdone
在 osd 启动之后, ceph 集群会自动执行数据迁移及数据平衡
[root@tt-ceph-048144 ceph]# ceph -w cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 1637 pgs degraded; 45 pgs recovering; 1592 pgs recovery_wait; 494 pgs stuck unclean; recovery 9690/36498 objects degraded (26.549%); 1 mons down, quorum 0,1 tt-ceph-048144,tt-ceph-048146 monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 28, quorum 0,1 tt-ceph-048144,tt-ceph-048146 osdmap e287: 36 osds: 30 up, 30 in pgmap v7407: 10304 pgs, 2 pools, 48041 MB data, 12166 objects 178 GB used, 720 GB / 899 GB avail 9690/36498 objects degraded (26.549%) 8667 active+clean 1592 active+recovery_wait+degraded 45 active+recovering+degradedrecovery io 55052 kB/s, 13 objects/s再参考
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 850 pgs degraded; 9 pgs peering; 48 pgs recovering; 800 pgs recovery_wait; 1 pgs stuck inactive; 261 pgs stuck unclean; recovery 5158/36498 objects degraded (14.132%) monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 36, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e291: 36 osds: 33 up, 33 in pgmap v7412: 10304 pgs, 2 pools, 48041 MB data, 12166 objects 180 GB used, 718 GB / 899 GB avail 5158/36498 objects degraded (14.132%) 2 active+degraded 9 peering 9445 active+clean 800 active+recovery_wait+degraded 48 active+recovering+degradedrecovery io 272 MB/s, 69 objects/s再次参考
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 2236 pgs degraded; 167 pgs peering; 18 pgs recovering; 1793 pgs recovery_wait; 104 pgs stuck inactive; 737 pgs stuck unclean; recovery 11298/36495 objects degraded (30.958%); 1 mons down, quorum 0,2 tt-ceph-048144,tt-ceph-048148 monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 50, quorum 0,2 tt-ceph-048144,tt-ceph-048148 osdmap e297: 36 osds: 35 up, 35 in pgmap v7419: 10304 pgs, 2 pools, 48037 MB data, 12165 objects 181 GB used, 867 GB / 1049 GB avail 11298/36495 objects degraded (30.958%) 73 inactive 143 degraded 282 active+degraded 166 peering 7828 active+clean 1 remapped+peering 1793 active+recovery_wait+degraded 18 active+recovering+degradedrecovery io 11214 kB/s, 2 objects/s约莫几分钟后, 数据同步完成
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 66, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e302: 36 osds: 36 up, 36 in pgmap v7460: 10304 pgs, 2 pools, 48041 MB data, 12166 objects 187 GB used, 892 GB / 1079 GB avail 10304 active+clean参考之前容量
[root@tt-ceph-048144 ceph]# ceph dfGLOBAL: SIZE AVAIL RAW USED %RAW USED 899G 724G 175G 19.50POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 235G 0 volumes 1 48041M 5.22 235G 12166
参考扩容后容量
[root@tt-ceph-048150 tmp]# ceph dfGLOBAL: SIZE AVAIL RAW USED %RAW USED 1049G 867G 181G 17.34POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 287G 0 volumes 1 48037M 4.47 287G 12165
参考新机器的磁盘容量
[root@tt-ceph-048150 /]# df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/centos-root 36G 3.8G 32G 11% /devtmpfs 1.9G 0 1.9G 0% /devtmpfs 1.9G 0 1.9G 0% /dev/shmtmpfs 1.9G 8.6M 1.9G 1% /runtmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup/dev/sda1 497M 96M 401M 20% /boot/dev/sdb1 30G 4.6G 26G 16% /var/lib/ceph/osd/ceph-30/dev/sdc1 30G 5.0G 26G 17% /var/lib/ceph/osd/ceph-31/dev/sdd1 30G 5.1G 25G 17% /var/lib/ceph/osd/ceph-32/dev/sde1 30G 5.0G 26G 17% /var/lib/ceph/osd/ceph-33/dev/sdf1 30G 4.7G 26G 16% /var/lib/ceph/osd/ceph-34/dev/sdg1 30G 5.0G 26G 17% /var/lib/ceph/osd/ceph-35
参考其他机器磁盘容量
[root@tt-ceph-048144 ceph]# df -h文件系统 容量 已用 可用 已用% 挂载点/dev/mapper/centos-root 36G 12G 25G 32% /devtmpfs 1.9G 0 1.9G 0% /devtmpfs 1.9G 0 1.9G 0% /dev/shmtmpfs 1.9G 8.7M 1.9G 1% /runtmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup/dev/sdd1 30G 4.8G 26G 16% /var/lib/ceph/osd/ceph-2/dev/sdf1 30G 4.9G 26G 17% /var/lib/ceph/osd/ceph-4/dev/sdb1 30G 5.1G 25G 17% /var/lib/ceph/osd/ceph-0/dev/sde1 30G 4.9G 26G 17% /var/lib/ceph/osd/ceph-3/dev/sdc1 30G 5.5G 25G 19% /var/lib/ceph/osd/ceph-1
6. osd 灾难性破坏与恢复
目标, 破坏一个 osd 对应的所有数据, 尝试恢复集群完整性
当前的 osd 环境正常,
[root@tt-ceph-048150 ~]# df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/centos-root 36G 3.8G 32G 11% /devtmpfs 1.9G 0 1.9G 0% /devtmpfs 1.9G 0 1.9G 0% /dev/shmtmpfs 1.9G 8.7M 1.9G 1% /runtmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup/dev/sda1 497M 96M 401M 20% /boot/dev/sdb1 30G 4.6G 26G 16% /var/lib/ceph/osd/ceph-30/dev/sdc1 30G 5.0G 26G 17% /var/lib/ceph/osd/ceph-31/dev/sdd1 30G 5.0G 26G 17% /var/lib/ceph/osd/ceph-32/dev/sde1 30G 4.9G 26G 17% /var/lib/ceph/osd/ceph-33 <- 计划需要破坏的对象/dev/sdf1 30G 4.7G 26G 16% /var/lib/ceph/osd/ceph-34/dev/sdg1 30G 5.0G 26G 17% /var/lib/ceph/osd/ceph-35
[root@tt-ceph-048144 ~]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 66, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e302: 36 osds: 36 up, 36 in pgmap v7570: 10304 pgs, 2 pools, 48041 MB data, 12166 objects 178 GB used, 900 GB / 1079 GB avail 10304 active+clean破坏 osd 操作
[root@tt-ceph-048150 ~]# kill -9 14088[root@tt-ceph-048150 ~]# umount /var/lib/ceph/osd/ceph-33
观察当前 osd 环境, 发现已经遭到破坏
[root@tt-ceph-048144 ~]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 805 pgs degraded; 779 pgs stuck unclean; 805 pgs undersized; recovery 1054/38058 objects degraded (2.769%); 1/36 in osds are down monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 66, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e304: 36 osds: 35 up, 36 in pgmap v7783: 10304 pgs, 2 pools, 50110 MB data, 12686 objects 187 GB used, 891 GB / 1079 GB avail 1054/38058 objects degraded (2.769%) 9499 active+clean 805 active+undersized+degraded client io 8794 kB/s rd, 40401 kB/s wr, 90 op/s[root@tt-ceph-048144 ~]# ceph osd tree -8 6 host tt-ceph-04815030 1 osd.30 up 131 1 osd.31 up 132 1 osd.32 up 133 1 osd.33 down 034 1 osd.34 up 135 1 osd.35 up 1
恢复 osd 过程
初始化磁盘
[root@tt-ceph-048150 ~]# dd if=/dev/zero of=/dev/sde1 bs=1M count=2020+0 records in20+0 records out20971520 bytes (21 MB) copied, 0.0192865 s, 1.1 GB/s[root@tt-ceph-048150 ~]# mkfs.xfs /dev/sde1meta-data=/dev/sde1 isize=256 agcount=4, agsize=1966016 blks = sectsz=512 attr=2, projid32bit=1 = crc=0data = bsize=4096 blocks=7864064, imaxpct=25 = sunit=0 swidth=0 blksnaming =version 2 bsize=4096 ascii-ci=0 ftype=0log =internal log bsize=4096 blocks=3839, version=2 = sectsz=512 sunit=0 blks, lazy-count=1realtime =none extsz=4096 blocks=0, rtextents=0 [root@tt-ceph-048150 ~]# mount /dev/sde1 /var/lib/ceph/osd/ceph-33
初始化 osd
[root@tt-ceph-048150 ~]# ceph-osd -i 33 --mkfs --mkkey --osd-uuid 60391f4d-734f-425d-89b2-1fcb26c876c8[root@tt-ceph-048150 ~]# ls /var/lib/ceph/osd/ceph-33/ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
授权会在初始化时候自动完成 检查授权 key 是否正确
[root@tt-ceph-048144 ~]# ceph auth listosd.33 key: AQBC1sFVSGx/JhAASmGkmXgJKXt7yiHaNPNlHQ== caps: [mon] allow profile osd caps: [osd] allow *
检查 osd 初始化后的 keyring
[root@tt-ceph-048150 ~]# cat /var/lib/ceph/osd/ceph-33/keyring[osd.33]key = AQDPD8NVMMQLNxAAPtYAXbqycCekTVwvavuUSA==
注意: 加入 key 不对, 则需要手动修改, 否则会遇到下面的报错
[root@tt-ceph-048150 ~]# /etc/init.d/ceph start osd.33=== osd.33 ===Error connecting to cluster: PermissionError2015-08-06 15:48:32.869373 7fa4cee30700 0 librados: osd.33 authentication error (1) Operation not permittedfailed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.33 --keyring=/var/lib/ceph/osd/ceph-33/keyring osd crush create-or-move -- 33 0.03 host=tt-ceph-048150 root=default'
启动 osd
[root@tt-ceph-048150 ~]# touch /var/lib/ceph/osd/ceph-33/sysvinit[root@tt-ceph-048150 ~]# /etc/init.d/ceph start osd.33=== osd.33 ===create-or-move updated item name 'osd.33' weight 0.03 at location {host=tt-ceph-048150,root=default} to crush mapStarting Ceph osd.33 on tt-ceph-048150...Running as unit run-19821.service.参考 ceph 集群监控
[root@tt-ceph-048144 ~]# ceph osd tree-8 6 host tt-ceph-04815030 1 osd.30 up 131 1 osd.31 up 132 1 osd.32 up 133 1 osd.33 up 134 1 osd.34 up 135 1 osd.35 up 1
osd 启动后, 数据自动执行恢复
[root@tt-ceph-048150 ~]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 36 pgs degraded; 3 pgs recovering; 36 pgs stuck unclean; recovery 261/45192 objects degraded (0.578%) monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 72, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e308: 36 osds: 36 up, 36 in pgmap v8838: 10304 pgs, 2 pools, 59531 MB data, 15064 objects 215 GB used, 863 GB / 1079 GB avail 261/45192 objects degraded (0.578%) 33 active+degraded 10268 active+clean 3 active+recovering+degradedrecovery io 191 MB/s, 48 objects/s[root@tt-ceph-048150 ~]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 72, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e308: 36 osds: 36 up, 36 in pgmap v8856: 10304 pgs, 2 pools, 59531 MB data, 15064 objects 215 GB used, 863 GB / 1079 GB avail 10304 active+clean7. 停电故障模拟
两个节点停电故障模拟
在集群环境正常状态, 为两个节点作停电处理
[root@tt-ceph-048144 /]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 76, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e400: 36 osds: 36 up, 36 in pgmap v10445: 10304 pgs, 2 pools, 59531 MB data, 15064 objects 212 GB used, 867 GB / 1079 GB avail 10304 active+clean client io 29033 kB/s rd, 6084 B/s wr, 236 op/s停电时, 涉及 1 个 mon 节点, 11 个 osd 节点
[root@tt-ceph-048144 /]# ceph osd tree# id weight type name up/down reweight-1 36 root default-2 5 host tt-ceph-0481440 1 osd.0 up 11 1 osd.1 up 12 1 osd.2 up 13 1 osd.3 up 14 1 osd.4 up 1-3 5 host tt-ceph-0481455 1 osd.5 up 16 1 osd.6 up 17 1 osd.7 up 18 1 osd.8 up 19 1 osd.9 up 1-4 5 host tt-ceph-04814610 1 osd.10 up 111 1 osd.11 up 112 1 osd.12 up 113 1 osd.13 up 114 1 osd.14 up 1-5 5 host tt-ceph-04814715 1 osd.15 up 116 1 osd.16 up 117 1 osd.17 up 118 1 osd.18 up 119 1 osd.19 up 1-6 5 host tt-ceph-04814820 1 osd.20 down 121 1 osd.21 down 122 1 osd.22 down 123 1 osd.23 down 124 1 osd.24 down 1-7 5 host tt-ceph-04814925 1 osd.25 up 126 1 osd.26 up 127 1 osd.27 up 128 1 osd.28 up 129 1 osd.29 up 1-8 6 host tt-ceph-04815030 1 osd.30 down 131 1 osd.31 down 132 1 osd.32 down 133 1 osd.33 down 134 1 osd.34 down 135 1 osd.35 down 1
参见停电后的集群状态
故障自动修复, 数据复制没有被中断
[root@tt-ceph-048144 /]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 7614 pgs degraded; 7470 pgs stuck unclean; 7614 pgs undersized; recovery 13695/45744 objects degraded (29.938%); 11/36 in osds are down; 1 mons down, quorum 0,1 tt-ceph-048144,tt-ceph-048146 monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 78, quorum 0,1 tt-ceph-048144,tt-ceph-048146 osdmap e404: 36 osds: 25 up, 36 in pgmap v10492: 10304 pgs, 2 pools, 60265 MB data, 15248 objects 216 GB used, 863 GB / 1079 GB avail 13695/45744 objects degraded (29.938%) 7614 active+undersized+degraded 2690 active+clean client io 11370 kB/s rd, 4902 kB/s wr, 95 op/s重新启动主机, 并重新启动 mon , osd 服务
再次参考 ceph 集群状态
[root@tt-ceph-048150 ~]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 4304 pgs degraded; 4247 pgs stuck unclean; 4304 pgs undersized; recovery 6765/47271 objects degraded (14.311%); 5/36 in osds are down monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 80, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e413: 36 osds: 31 up, 36 in pgmap v10584: 10304 pgs, 2 pools, 62283 MB data, 15757 objects 223 GB used, 855 GB / 1079 GB avail 6765/47271 objects degraded (14.311%) 4304 active+undersized+degraded 6000 active+clean client io 19230 kB/s rd, 155 op/s[root@tt-ceph-048150 ~]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 281 pgs stale monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 80, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e422: 36 osds: 36 up, 36 in pgmap v10614: 10304 pgs, 2 pools, 62634 MB data, 15846 objects 227 GB used, 852 GB / 1079 GB avail 281 stale+active+clean 10023 active+clean client io 2283 kB/s rd, 11928 B/s wr, 20 op/s故障自动修复, 数据复制没有被中断
[root@tt-ceph-048150 ~]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 80, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e422: 36 osds: 36 up, 36 in pgmap v10621: 10304 pgs, 2 pools, 62766 MB data, 15879 objects 227 GB used, 851 GB / 1079 GB avail 10304 active+clean client io 12088 kB/s rd, 4543 kB/s wr, 101 op/s三节点停电测试
[root@tt-ceph-048144 /]# ceph osd tree^[[A# id weight type name up/down reweight-1 36 root default-2 5 host tt-ceph-0481440 1 osd.0 up 11 1 osd.1 up 12 1 osd.2 up 13 1 osd.3 up 14 1 osd.4 up 1-3 5 host tt-ceph-0481455 1 osd.5 up 16 1 osd.6 up 17 1 osd.7 up 18 1 osd.8 up 19 1 osd.9 up 1-4 5 host tt-ceph-04814610 1 osd.10 up 111 1 osd.11 up 112 1 osd.12 up 113 1 osd.13 up 114 1 osd.14 up 1-5 5 host tt-ceph-04814715 1 osd.15 down 116 1 osd.16 down 117 1 osd.17 down 118 1 osd.18 down 119 1 osd.19 down 1-6 5 host tt-ceph-04814820 1 osd.20 down 121 1 osd.21 down 122 1 osd.22 down 123 1 osd.23 down 124 1 osd.24 down 1-7 5 host tt-ceph-04814925 1 osd.25 up 126 1 osd.26 up 127 1 osd.27 up 128 1 osd.28 up 129 1 osd.29 up 1-8 6 host tt-ceph-04815030 1 osd.30 down 131 1 osd.31 down 132 1 osd.32 down 133 1 osd.33 down 134 1 osd.34 down 135 1 osd.35 down 1
参考停电期间的状态 (关闭 1 MON 节点, 16 OSD 节点)
[root@tt-ceph-048144 /]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 9232 pgs degraded; 321 pgs stale; 7797 pgs stuck unclean; 9232 pgs undersized; recovery 21510/49914 objects degraded (43.094%); 16/36 in osds are down; 1 mons down, quorum 0,1 tt-ceph-048144,tt-ceph-048146 monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 82, quorum 0,1 tt-ceph-048144,tt-ceph-048146 osdmap e426: 36 osds: 20 up, 36 in pgmap v10814: 10304 pgs, 2 pools, 65782 MB data, 16638 objects 235 GB used, 844 GB / 1079 GB avail 21510/49914 objects degraded (43.094%) 8911 active+undersized+degraded 1072 active+clean 321 stale+active+undersized+degraded client io 3291 kB/s rd, 2477 B/s wr, 27 op/s重新启动三个节点, 启动 mon, osd 服务
[root@tt-ceph-048148 ~]# ceph osd tree# id weight type name up/down reweight-1 36 root default-2 5 host tt-ceph-0481440 1 osd.0 up 11 1 osd.1 up 12 1 osd.2 up 13 1 osd.3 up 14 1 osd.4 up 1-3 5 host tt-ceph-0481455 1 osd.5 up 16 1 osd.6 up 17 1 osd.7 up 18 1 osd.8 up 19 1 osd.9 up 1-4 5 host tt-ceph-04814610 1 osd.10 up 111 1 osd.11 up 112 1 osd.12 up 113 1 osd.13 up 114 1 osd.14 up 1-5 5 host tt-ceph-04814715 1 osd.15 up 116 1 osd.16 up 117 1 osd.17 up 118 1 osd.18 up 119 1 osd.19 up 1-6 5 host tt-ceph-04814820 1 osd.20 up 121 1 osd.21 up 122 1 osd.22 up 123 1 osd.23 up 124 1 osd.24 up 1-7 5 host tt-ceph-04814925 1 osd.25 up 126 1 osd.26 up 127 1 osd.27 up 128 1 osd.28 up 129 1 osd.29 up 1-8 6 host tt-ceph-04815030 1 osd.30 up 131 1 osd.31 up 132 1 osd.32 up 133 1 osd.33 up 134 1 osd.34 up 135 1 osd.35 up 1
参考集群状态
[root@tt-ceph-048148 ~]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_WARN 2750 pgs degraded; 207 pgs down; 207 pgs peering; 207 pgs stuck inactive; 1822 pgs stuck unclean; 2749 pgs undersized; recovery 4525/49914 objects degraded (9.066%) monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 84, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e443: 36 osds: 36 up, 36 in pgmap v10843: 10304 pgs, 2 pools, 65782 MB data, 16638 objects 232 GB used, 846 GB / 1079 GB avail 4525/49914 objects degraded (9.066%) 2749 active+undersized+degraded 1 active+degraded 7347 active+clean 207 down+peeringrecovery io 1116 MB/s, 283 objects/s client io 1247 MB/s rd, 2683 kB/s wr, 9946 op/s集群会被自动修复成功
[root@tt-ceph-048148 ~]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 84, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e443: 36 osds: 36 up, 36 in pgmap v10855: 10304 pgs, 2 pools, 65874 MB data, 16662 objects 233 GB used, 845 GB / 1079 GB avail 10304 active+clean client io 3240 kB/s rd, 26 op/s整个 CEPH 集群停电测试
在所有 ceph 节点关闭后, 重启启动主机, 分别启动 MON OSD 服务后, 集群会自动恢复正常, 客户可继续访问, 读写云盘
[root@tt-ceph-048144 ceph]# ceph -s cluster 60391f4d-734f-425d-89b2-1fcb26c876c8 health HEALTH_OK monmap e1: 3 mons at {tt-ceph-048144=192.168.48.144:6789/0,tt-ceph-048146=192.168.48.146:6789/0,tt-ceph-048148=192.168.48.148:6789/0}, election epoch 90, quorum 0,1,2 tt-ceph-048144,tt-ceph-048146,tt-ceph-048148 osdmap e474: 36 osds: 36 up, 36 in pgmap v10961: 10304 pgs, 2 pools, 66864 MB data, 16911 objects 239 GB used, 840 GB / 1079 GB avail 10304 active+clean client io 2930 kB/s rd, 2929 B/s wr, 24 op/s 0 0
- osd 故障测试
- OSD
- OSD
- OSD
- kubernetes中的中间件集群故障测试
- DM6446 OSD
- DM6446 OSD
- 关于OSD
- 中文OSD
- MAX7456 OSD
- OSD::ms_fast_dispatch
- 海思OSD
- 英国ABI电路板故障测试系统8000
- ceph 可靠性测试 单故障域故障测试、单磁盘故障测试、单节点故障测试、单机柜故障测试、故障数据重构测试
- ceph (luminous 版) journal disk 故障测试
- ceph (luminous 版) data disk 故障测试
- OSD-04002问题
- OSD终结篇
- Jcrop（头像裁剪工具）
- HDU 2072 单词数
- MPMoviePlayerController 添加自定义控件
- 使用memcache存储数据
- SAP Basis: 从SAP GUI杀死进程
- osd 故障测试
- POJ2531.Network Saboteur
- MATLAB连接MYSQL搞数据
- 页面静态化
- 在NVIDIA GPU 上开发 openCL
- EmguCV获取kinect的彩色帧和红外图
- CLR程序集-怎么把一个dll注册到SQL SEVER
- jsp 中 mate 标签的作用
- 【华为oj】截取字符串，汉字不能截一半
