Ceph实战入门系列（一）——三节点Ceph集群的安装与部署
来源：互联网 发布：游戏程序员面试 编辑：程序博客网 时间：2024/04/30 19:10
配置环境
环境说明
- 在Win7下安装vmware软件，用CentOS-7.2创建一台centos7.2-mini虚拟机，安装完后把内存改为512MB就足够了，如果是用来实践的话。
- 基于上面的虚拟机链接克隆3台虚拟机
修改hostname
# 分别在3个host上执行[root@localhost ~]# hostnamectl set-hostname ceph6[root@localhost ~]# hostnamectl set-hostname ceph7[root@localhost ~]# hostnamectl set-hostname ceph8配置解析
# 全体都有,并测试网络是否连通[root@ceph8 ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.103.139 ceph6192.168.103.140 ceph7192.168.103.138 ceph8配置防火墙启用端口
# 全体都有[root@ceph6 ~]# firewall-cmd --zone=public --add-port=6789/tcp --permanent success[root@ceph6 ~]# firewall-cmd --zone=public --add-port=6800-7100/tcp --permanent success[root@ceph6 ~]# firewall-cmd --reloadsuccess[root@ceph6 ~]# firewall-cmd --zone=public --list-all public (default, active) interfaces: eno16777736 sources: services: dhcpv6-client ssh ports: 6789/tcp 6800-7100/tcp masquerade: no forward-ports: icmp-blocks: rich rules: 禁用SELINUX
# 全体都有[root@ceph6 ~]# setenforce 0[root@ceph6 ~]# vim /etc/selinux/config [root@ceph6 ~]# cat /etc/selinux/config | grep disabled# disabled - No SELinux policy is loaded.SELINUX=disabled配置网络
# 全体都有；公司内部开发网需要单独设置[root@ceph6 ~]# cat /etc/yum.conf [main]cachedir=/var/cache/yum/$basearch/$releaseverkeepcache=0debuglevel=2logfile=/var/log/yum.logexactarch=1obsoletes=1gpgcheck=1plugins=1installonly_limit=5bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://bugs.centos.org/bug_report_page.php?category=yumdistroverpkg=centos-releaseProxy=http://dev-proxy.oa.com:8080/[root@ceph6 ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain610.14.87.100 dev-proxy.oa.com设置全局代理
# 全体都有；公司网络限制，需要设置全局代理，需要重启主机（poweroff命令，再开机）http_proxy=dev-proxy.oa.com:8080https_proxy=dev-proxy.oa.com:8080export http_proxy https_proxy安装配置ntp服务
# 全体都有， 10.14.0.131是公司的ntp server[root@ceph6 ~]# cat /etc/ntp.conf | grep server | grep -v '#'server 10.14.0.131wget下载rpm；代理
# wget代理下载rpm，发现不能install，不知道原因[root@ceph6 ~]# wget http://dev-proxy.oa.com:8080/ http://download.ceph.com/rpm-giant/el7/noarch/ceph-release-1-0.el7.noarch.rpm -e http_proxy=dev-proxy.oa.com:8080/Ceph安装与部署
安装部署工具
[root@ceph6 ~]# yum install -y ceph-deploy初始化monitor
# 在ceph6上操作[root@ceph6 ~]# mkdir /etc/ceph[root@ceph6 ~]# cd /etc/ceph/[root@ceph6 ceph]# ceph-deploy new ceph6[root@ceph6 ceph]# lsceph.conf ceph.log ceph.mon.keyring免密码登录
# 在ceph6上操作[root@ceph6 ~]# ssh-keygen [root@ceph6 ~]# ssh-copy-id root@ceph7[root@ceph6 ~]# ssh-copy-id root@ceph8远程安装部署
# 网络问题;通过第一步的设置全局代理并未解决[root@ceph6 ~]# ceph-deploy install ceph6 ceph7 ceph8# 省略[ceph6][DEBUG ] [ceph6][DEBUG ] Complete![ceph6][DEBUG ] Configure Yum priorities to include obsoletes[ceph6][WARNIN] check_obsoletes has been enabled for Yum priorities plugin[ceph6][INFO ] Running command: rpm --import https://download.ceph.com/keys/release.asc[ceph6][WARNIN] curl: (7) Failed to connect to 2607:f298:6050:51f3:f816:3eff:fe71:9135: Network is unreachable[ceph6][WARNIN] error: https://download.ceph.com/keys/release.asc: import read failed(2).[ceph6][ERROR ] RuntimeError: command returned non-zero exit status: 1[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: rpm --import https://download.ceph.com/keys/release.asc独立安装部署
[root@ceph7 ~]# ceph-deploy install ceph7[root@ceph8 ~]# ceph-deploy install ceph8搭建集群
创建OSD
# 列出ceph6上所有的可用磁盘[root@ceph6 ceph]# ceph-deploy disk list ceph6# disk zap 子命令删除现有分区表和磁盘内容[root@ceph6 ceph]# ceph-deploy disk zap ceph6:sdb ceph6:sdc ceph6:sdd[root@ceph6 ceph]# ceph-deploy disk zap ceph7:sdb ceph7:sdc ceph7:sdd[root@ceph6 ceph]# ceph-deploy disk zap ceph8:sdb ceph8:sdc ceph8:sdd# osd create 子命令首先准备磁盘（用xfs文件系统格式化磁盘），然后激活磁盘的第一、第二个分区，作为数据分区和日志分区[root@ceph6 ceph]# ceph-deploy osd create ceph6:sdb ceph6:sdc ceph6:sdd[root@ceph6 ceph]# ceph-deploy osd create ceph7:sdb ceph7:sdc ceph7:sdd[root@ceph6 ceph]# ceph-deploy osd create ceph8:sdb ceph8:sdc ceph8:sdd# 检查ceph的状态[root@ceph6 ceph]# ceph -s cluster 8ea4fa79-3b6b-4de3-8cfb-a0922d6827c5 health HEALTH_WARN too few PGs per OSD (21 < min 30) monmap e1: 1 mons at {ceph6=192.168.103.139:6789/0} election epoch 3, quorum 0 ceph6 osdmap e43: 9 osds: 9 up, 9 in flags sortbitwise pgmap v95: 64 pgs, 1 pools, 0 bytes data, 0 objects 307 MB used, 134 GB / 134 GB avail 64 active+clean# 解决HEALTH_WARN[root@ceph6 ceph]# ceph osd pool set rbd pg_num 128[root@ceph6 ceph]# ceph osd pool set rbd pg_num 128[root@ceph6 ceph]# ceph -s cluster 8ea4fa79-3b6b-4de3-8cfb-a0922d6827c5 health HEALTH_OK monmap e1: 1 mons at {ceph6=192.168.103.139:6789/0} election epoch 3, quorum 0 ceph6 osdmap e48: 9 osds: 9 up, 9 in flags sortbitwise pgmap v130: 128 pgs, 1 pools, 0 bytes data, 0 objects 309 MB used, 134 GB / 134 GB avail 128 active+clean部署奇数个数的monitor
# 将公共网络地址添加到配置文件[root@ceph6 ceph]# cat /etc/ceph/ceph.conf [global]fsid = 8ea4fa79-3b6b-4de3-8cfb-a0922d6827c5mon_initial_members = ceph6,ceph7,ceph8mon_host = 192.168.103.139,192.168.103.140,192.168.103.138auth_cluster_required = cephxauth_service_required = cephxauth_client_required = cephxpublic network = 192.168.103.0/24# 使用ceph-deploy命令在ceph7和ceph8上创建一个monitor[root@ceph6 ceph]# ceph-deploy mon create ceph7 --overwrite-conf[root@ceph6 ceph]# ceph-deploy --overwrite-conf mon create ceph8实践使用ceph集群
检查Ceph的状态
[root@ceph6 ceph]# ceph -s cluster 8ea4fa79-3b6b-4de3-8cfb-a0922d6827c5 health HEALTH_OK monmap e3: 3 mons at {ceph6=192.168.103.139:6789/0,ceph7=192.168.103.140:6789/0,ceph8=192.168.103.138:6789/0} election epoch 8, quorum 0,1,2 ceph8,ceph6,ceph7 osdmap e48: 9 osds: 9 up, 9 in flags sortbitwise pgmap v130: 128 pgs, 1 pools, 0 bytes data, 0 objects 309 MB used, 134 GB / 134 GB avail 128 active+clean[root@ceph6 ceph]# ceph status cluster 8ea4fa79-3b6b-4de3-8cfb-a0922d6827c5 health HEALTH_OK monmap e3: 3 mons at {ceph6=192.168.103.139:6789/0,ceph7=192.168.103.140:6789/0,ceph8=192.168.103.138:6789/0} election epoch 8, quorum 0,1,2 ceph8,ceph6,ceph7 osdmap e48: 9 osds: 9 up, 9 in flags sortbitwise pgmap v130: 128 pgs, 1 pools, 0 bytes data, 0 objects 309 MB used, 134 GB / 134 GB avail 128 active+clean观察集群健康状态
[root@ceph6 ceph]# ceph -w cluster 8ea4fa79-3b6b-4de3-8cfb-a0922d6827c5 health HEALTH_OK monmap e3: 3 mons at {ceph6=192.168.103.139:6789/0,ceph7=192.168.103.140:6789/0,ceph8=192.168.103.138:6789/0} election epoch 8, quorum 0,1,2 ceph8,ceph6,ceph7 osdmap e48: 9 osds: 9 up, 9 in flags sortbitwise pgmap v130: 128 pgs, 1 pools, 0 bytes data, 0 objects 309 MB used, 134 GB / 134 GB avail 128 active+clean2016-11-09 10:01:39.675477 mon.0 [INF] from='client.4174 :/0' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.bootstrap-mds", "caps": ["mon", "allow profile bootstrap-mds"]}]: dispatch检查Ceph monitor仲裁状态
[root@ceph6 ceph]# ceph quorum_status --format json-pretty{ "election_epoch": 8, "quorum": [ 0, 1, 2 ], "quorum_names": [ "ceph8", "ceph6", "ceph7" ], "quorum_leader_name": "ceph8", "monmap": { "epoch": 3, "fsid": "8ea4fa79-3b6b-4de3-8cfb-a0922d6827c5", "modified": "2016-11-09 10:01:31.732730", "created": "2016-11-08 22:36:48.791105", "mons": [ { "rank": 0, "name": "ceph8", "addr": "192.168.103.138:6789\/0" }, { "rank": 1, "name": "ceph6", "addr": "192.168.103.139:6789\/0" }, { "rank": 2, "name": "ceph7", "addr": "192.168.103.140:6789\/0" } ] }}导出Ceph monitor信息
[root@ceph6 ceph]# ceph mon dumpdumped monmap epoch 3epoch 3fsid 8ea4fa79-3b6b-4de3-8cfb-a0922d6827c5last_changed 2016-11-09 10:01:31.732730created 2016-11-08 22:36:48.7911050: 192.168.103.138:6789/0 mon.ceph81: 192.168.103.139:6789/0 mon.ceph62: 192.168.103.140:6789/0 mon.ceph7检查集群使用状态
[root@ceph6 ceph]# ceph dfGLOBAL: SIZE AVAIL RAW USED %RAW USED 134G 134G 309M 0.22 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 45942M 0 检查Ceph monitor、OSD和PG（配置组）状态
[root@ceph6 ceph]# ceph mon state3: 3 mons at {ceph6=192.168.103.139:6789/0,ceph7=192.168.103.140:6789/0,ceph8=192.168.103.138:6789/0}, election epoch 8, quorum 0,1,2 ceph8,ceph6,ceph7[root@ceph6 ceph]# ceph osd stat osdmap e48: 9 osds: 9 up, 9 in flags sortbitwise[root@ceph6 ceph]# ceph pg statv130: 128 pgs: 128 active+clean; 0 bytes data, 309 MB used, 134 GB / 134 GB avail列表PG
[root@ceph6 ceph]# ceph pg dump列表Ceph存储池
[root@ceph6 ceph]# ceph osd lspools0 rbd,检查OSD的CRUSH map
[root@ceph6 ceph]# ceph osd treeID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 0.13129 root default -2 0.04376 host ceph6 0 0.01459 osd.0 up 1.00000 1.00000 1 0.01459 osd.1 up 1.00000 1.00000 2 0.01459 osd.2 up 1.00000 1.00000 -3 0.04376 host ceph7 3 0.01459 osd.3 up 1.00000 1.00000 4 0.01459 osd.4 up 1.00000 1.00000 5 0.01459 osd.5 up 1.00000 1.00000 -4 0.04376 host ceph8 6 0.01459 osd.6 up 1.00000 1.00000 7 0.01459 osd.7 up 1.00000 1.00000 8 0.01459 osd.8 up 1.00000 1.00000 列表集群的认证密钥
[root@ceph6 ceph]# ceph auth list 0 0
- Ceph实战入门系列（一）——三节点Ceph集群的安装与部署
- Ceph学习1——Ubuntu12.04手动安装Ceph&&部署Ceph集群
- Ceph实战入门系列（二）——块设备
- Ceph快速安装：Ceph-deploy 在Ubuntu 14.04 Server上部署三节点安装
- Ceph快速安装：Ceph-deploy 在Ubuntu 14.04 Server上部署三节点安装
- Ceph部署（一）集群搭建
- ceph入门----ceph安装
- centos7（三节点）搭建ceph环境
- ceph-deploy部署ceph集群
- Hadoop集群（三节点）安装与部署
- Ceph实战入门系列（三）——块设备-OpenStack
- “Ceph浅析”系列之(五)——Ceph与OpenStack
- “Ceph浅析”系列之六——Ceph与OpenStack
- 用ceph-deploy安装ceph并部署集群
- 20170807ceph入门(一)——ceph中rados对象创建以及文件上传与下载
- 【Ceph学习之一】Centos7上部署Ceph存储集群以及CephFS的安装
- “Ceph浅析”系列之四——Ceph的结构
- “Ceph浅析”系列之三——Ceph的结构
- APP开发实战145-Enum枚举变量的处理
- 【Windows】如何关闭Windows Defender？
- Technical Artist的不归路 —— 创建游戏世界前团队交流之我见
- Android 5.0 源码 USB插拔事件监听（代码摘自google）
- CentOS 7安装Gnome GUI 图形界面
- Ceph实战入门系列（一）——三节点Ceph集群的安装与部署
- Smartbi的安装部署
- oracle on commit delete rows 和on commit preserve rows的区别
- 折纸问题 。。。
- sql查询某两列（或者多列）的值相同，的所有记录
- HTML标签-marquee
- java 上传图片 二进制保存到 mysql 请求图片二进制转化为图片 cxf rest jax-rs
- Okio组件源码分析
- srvctl添加RAC备库DG资源
