Openstack 多后端配置--详解NFS作为Cinder的后端存储

材料：
1.现有一外置NFS存储，大小为300G，路径为：192.168.8.213:/test3/cinder
2.mitaka版本，已配置lvm后端存储，大小为20G

NFS后端存储本地挂载(不挂载也不影响)
挂载到本地目录

cd /rootmkdir nfssharemount -t nfs 192.168.8.213:/test3/cinder /root/nfsshare/ 

查看挂载结果

    [root@controllernode~(keystone_admin)]# df -h      Filesystem                           Size  Used Avail Use% Mounted on      /dev/sda1                             97G  4.8G  87G   6% /      tmpfs                                3.9G  4.0K 3.9G   1% /dev/shm      /srv/loopback-device/swift_loopback  1.9G  67M  1.8G   4% /srv/node/swift_loopback      192.168.8.213:/test3/cinder           300G 1.4G  298G   1% /root/nfsshare

需要注意的是，如果此时NFS服务器出现故障，或者是客户端不能连接到服务端，由于该命令要等待文件系统查找超时后才返回结果，所以该操作会变的很慢，该原理是用于所有针对文件系统的命令，例如 df,ls,cp等

Cinder节点NFS后端存储配置
①创建/etc/cinder/nfsshares文件（该文件的作用是列出所有可用的nfs存储），并编辑内容如下

192.168.8.213:/test3/cinder

设置配置文件的权限

[root@controllernode~]# chown root:cinder /etc/cinder/nfsshares[root@controllernode~]# chmod 0640 /etc/cinder/nfsshares  

②/etc/cinder/cinder.conf中配置cinder的volume 服务添加NFS：

即：enabled_backends = lvm,nfs

③并添加如下内容

即：

[nfs]volume_driver=cinder.volume.drivers.nfs.NfsDrivernfs_shares_config = /etc/cinder/nfs_sharesnfs_mount_point_base = /var/lib/cinder/mntvolume_backend_name = nfs

上述配置条目详细作用介绍：

【备注：添加③内容，可以用命令行执行添加–
修改/etc/cinder/cinder.conf中的值为/etc/cinder/nfsshares,可执行下面的命令
openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfsshares
配置cinder的volume使用的驱动，可使用以下命令
openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver
】

④重新启动服务
service openstack-cinder-volume restart

此时查看客户端上磁盘的情况

添加了最后一条记录

⑤创建一个虚拟机，网络硬盘，并将网络硬盘挂载到主机上
控制台：

虚拟机：

在虚拟机中挂载硬盘
先使用下面的命令格式化
mkfs.ext4 /dev/vdb

问题1：
挂载的过程中，nova/compute.log中出现以下异常

    2014-10-2312:23:28.193 1747 INFO urllib3.connectionpool [-] Starting new HTTP connection(1): 192.168.40.248      2014-10-2312:23:28.395 1747 WARNING nova.virt.libvirt.utils[req-5bf92b88-6d15-4c41-8ed7-3325fdea0dcf 5832a2295dc14de79522ee8b42e7daac9207105ae2ac4ef3bdf5dfe40d99fd8d] systool is not installed      2014-10-2312:23:28.449 1747 WARNING nova.virt.libvirt.utils[req-5bf92b88-6d15-4c41-8ed7-3325fdea0dcf 5832a2295dc14de79522ee8b42e7daac9207105ae2ac4ef3bdf5dfe40d99fd8d] systool is not installed      2014-10-2312:23:28.451 1747 INFO urllib3.connectionpool [-] Starting new HTTP connection(1): 192.168.40.248      2014-10-2312:23:28.960 1747 ERROR nova.virt.block_device[req-5bf92b88-6d15-4c41-8ed7-3325fdea0dcf 5832a2295dc14de79522ee8b42e7daac9207105ae2ac4ef3bdf5dfe40d99fd8d] [instance:eb1742c6-1e73-4656-b646-ca8442519e7a] Driver failed to attach volume a1862c54-0671-4cc5-9fce-5e5f8485c21fat /dev/vdb      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a] Traceback (most recent call last):      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File"/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line239, in attach      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    device_type=self['device_type'], encryption=encryption)      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line1267, in attach_volume      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    disk_dev)      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",line 68, in __exit__      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    six.reraise(self.type_, self.value, self.tb)      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line1254, in attach_volume      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    virt_dom.attachDeviceFlags(conf.to_xml(), flags)      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File "/usr/lib/python2.6/site-packages/eventlet/tpool.py",line 183, in doit      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    result = proxy_call(self._autowrap, f, *args, **kwargs)      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File "/usr/lib/python2.6/site-packages/eventlet/tpool.py",line 141, in proxy_call      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    rv = execute(f, *args, **kwargs)      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File     2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    six.reraise(c, e, tb)      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File "/usr/lib/python2.6/site-packages/eventlet/tpool.py",line 80, in tworker      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    rv = meth(*args, **kwargs)      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]  File "/usr/lib64/python2.6/site-packages/libvirt.py", line419, in attachDeviceFlags      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a]    if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed',dom=self)      2014-10-2312:23:28.960 1747 TRACE nova.virt.block_device [instance:eb1742c6-1e73-4656-b646-ca8442519e7a] libvirtError: internal error unable toexecute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1'could not be initialized

解决方法
这个错来自libvirt，做以下设置即可，先察看virt_use_nfs是off还是on
/usr/sbin/getsebool virt_use_nfs

如果是off，做以下设置
/usr/sbin/setsebool -P virt_use_nfs on

问题2：
云硬盘挂载过程中，nova/compute.log中出现异常：

2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4756, in _attach_volume2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     do_check_attach=False, do_driver_attach=True)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 54, in wrapped2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     ret_val = method(obj, context, *args, **kwargs)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 288, in attach2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     connector)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 279, in attach2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     device_type=self['device_type'], encryption=encryption)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1358, in attach_volume2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     self._disconnect_volume(connection_info, disk_dev)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     guest.attach_device(conf, persistent=True, live=live)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher libvirtError: Cannot access storage file '/var/lib/nova/mnt/82b29b840347978f288c9b93a4d5e849/volume-9b99cf0e-9381-48fb-8bdd-07a40efa80fa' (as uid:107, gid:107): No such file or directory2017-12-04 17:21:10.837 2806 ERROR oslo_messaging.rpc.dispatcher

解决方法
这个异常来自于/etc/nova/nova.conf配置的nfs_mount_point_base默认路径
修改或添加以下内容：
nfs_mount_point_base=/var/lib/cinder/mnt

==========NFS后端配置到此为止==========

Cinder节点多后端存储配置

下面是包含三个配置组的多后端示例（/etc/cinder/cinder.conf ）：

enable_backends =sata,sas,ssd[sata]volume_group=cinder-volumes-1volume_driver=cinder.volume.drivers.lvm.LVMISCSIDrivervolume_backend_name=sata[sas]volume_group=cinder-volumes-2volume_driver=cinder.volume.drivers.lvm.LVMISCSIDrivervolume_backend_name=sas[ssd]volume_group=cinder-volumes-3volume_driver=cinder.volume.drivers.lvm.LVMISCSIDrivervolume_backend_name=ssd

注意配置组名和后端名（volume_backend_name）没有关联，你可以取任何合法的字符串作为配置组名和后端名，而不用保持二者一致。

Cinder设置卷类型
可以为每个后端关联一个卷类型，创建卷的时候，调度器就能根据卷类型选择合适的后端来处理请求
下面的例子中创建一个名为sata的卷类型并与名为sata的后端关联

#> cinder type-create sata#> cinder type-key sata set volume_backend_name=sata

如果volume_backend_name指定的后端不存在，在创建卷的时候，过滤调度器将返回无法找到合适的后端的错误

Cinder创建卷
完成后端与卷类型的关联后，就可以创建卷了。当然你也可以不指定卷类型，调度器将使用默认的卷类型（由default_volume_type指定）.
下面的例子创建了一个大小为10G类型为sata的卷

#> cinder create-volume --volume-type sata --display_name sata1 10 

整个配置过程还是比较简单的，希望本文能给刚入门的你一点收获。

Cinder支持的存储后端

    * Blockbridge EPS    * Ceph RADOS Block Device (RBD)    * CloudByte volume driver    * Coho Data volume driver    * Dell EqualLogic volume driver    * Dell Storage Center Fibre Channel and iSCSI drivers    * Dot Hill AssuredSAN Fibre Channel and iSCSI drivers    * EMC ScaleIO Block Storage driver configuration    * EMC VMAX iSCSI and FC drivers    * EMC VNX driver    * EMC XtremIO Block Storage driver configuration    * Fujitsu ETERNUS DX driver    * GlusterFS driver    * HDS HNAS iSCSI and NFS driver    * Hitachi storage volume driver    * HPE 3PAR Fibre Channel and iSCSI drivers    * HPE LeftHand/StoreVirtual driver    * HP MSA Fibre Channel and iSCSI drivers    * Huawei volume driver    * IBM GPFS volume driver    * IBM Storwize family and SVC volume driver    * IBM XIV and DS8000 volume driver    * IBM FlashSystem volume driver    * ITRI DISCO volume driver    * Lenovo Fibre Channel and iSCSI drivers    * LVM    * NetApp unified driver    * Nimble Storage volume driver    * NexentaStor 4.x NFS and iSCSI drivers    * NexentaStor 5.x NFS and iSCSI drivers    * NexentaEdge iSCSI driver    * NFS driver    * ProphetStor Fibre Channel and iSCSI drivers    * Pure Storage iSCSI and Fibre Channel volume drivers    * Quobyte driver    * Scality SOFS driver    * Sheepdog driver    * SambaFS driver    * SolidFire    * Tintri    * Violin Memory 7000 Series FSP volume driver    * VMware VMDK driver    * Windows iSCSI volume driver    * X-IO volume driver    * Oracle ZFS Storage Appliance iSCSI driver    * Oracle ZFS Storage Appliance NFS driver

另外：
cinder支持的多后端：
http://docs.openstack.org/mitaka/config-reference/block-storage/volume-drivers.html

参考文献地址：
http://www.aboutyun.com/thread-9849-1-1.html
http://blog.csdn.net/lzw06061139/article/details/51276805