搭建CEPH测试环境

环境

硬件环境：
- 单机（虚拟机）
- CPU 8 cores
- MEM 32G
- DISK 3 x 100G

软件环境：
- Centos7 x64
- 数据盘 /dev/sdb1 /data1 xfs
- 数据盘 /dev/sdc1 /data2 xfs
- 系统接入互联网

CEPH版本
- ceph-deploy v1.5.37
- ceph version 10.2.7

CEPH 架构

CEPH统一地提供了对象、文件和块文件系统，如下图：

本文只搭建了Object Storage文件系统，未实现Block Device和Ceph FS文件系统。主要步骤如下：
- Monitor
- OSDs
- MDS
- RGW
- TEST（put & get file）

系统环境配置

Internet

vi /etc/sysconfig/network-scripts/ifcfg-ens33BOOTPROTO=staticONBOOT=yesIPADDR=192.168.3.125DNS1=192.168.3.1GATEWAY=192.168.3.1

Hostname

sed -i '/HOSTNAME/d' /etc/sysconfig/networkecho "HOSTNAME=cydb" >> /etc/sysconfig/networkcat /etc/sysconfig/networkecho "192.168.3.125 cydb" >> /etc/hostscat /etc/hostshostname cydbhostname -f

SSH

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsassh-copy-id root@cydb

Firewall

systemctl stop firewalld.servicesystemctl disable firewalld.service systemctl status firewalld.service

SELinux

setenforce 0sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/configcat /etc/selinux/configrebootsestatus

Ojbect文件系统

Ceph安装源

yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpmcat > /etc/yum.repos.d/ceph.repo << EOF[ceph-noarch]name=Ceph noarch packagesbaseurl=https://download.ceph.com/rpm-jewel/el7/noarchenabled=1gpgcheck=1type=rpm-mdgpgkey=https://download.ceph.com/keys/release.ascEOFcat /etc/yum.repos.d/ceph.repoyum clean allyum makecache

安装ceph-deploy

yum install -y ceph-deployceph-deploy --help

部署cluster

mkdir /opt/ceph-clustercd /opt/ceph-clusterceph-deploy new cydbecho "osd crush chooseleaf type = 0" >> ceph.confecho "osd pool default size = 1" >> ceph.confecho "osd journal size = 100" >> ceph.confcat ceph.conf

安装ceph包

yum remove -y ceph-release # 如果安装失败，则执行ceph-deploy install cydbceph --version

部署Monitor

配置文件定义了hostname

ceph-deploy mon create-initialceph mon statceph mon dump

添加OSDs

chown ceph:ceph /data1chown ceph:ceph /data2ceph-deploy osd prepare cydb:/data1ceph-deploy osd prepare cydb:/data2ceph-deploy osd activate cydb:/data1ceph-deploy osd activate cydb:/data2ceph osd statceph osd treeceph osd dump

推送配置文件和密钥文件

ceph-deploy admin cydbchmod +r /etc/ceph/ceph.client.admin.keyringceph status

部署MDS

ceph-deploy mds create cydbceph mds stat

创建RGW实例

ceph-deploy rgw create cydb

测试

# create poolrados mkpool data# put objectecho Test-data > testfile.txtcat testfile.txtrados put test-object-1 testfile.txt --pool=data# list poolrados -p data ls# lookup object locationceph osd map data test-object-1# get objectrados get test-object-1 testfile.get.txt --pool=datacat testfile.get.txt# del object from poolrados rm test-object-1 --pool=data

参考

CEPH官网 http://docs.ceph.com/docs/master/
Ubuntu 14.04 单机安装 CEPH http://www.cnblogs.com/YaoDD/p/5217578.html