OpenStack实践系列⑨云硬盘服务Cinder

OpenStack实践系列⑨云硬盘服务Cinder
八、cinder
8.1存储的三大分类

块存储：硬盘，磁盘阵列DAS，SAN存储
文件存储：nfs，GluserFS，Ceph（PB级分布式文件系统），MooserFS（缺点Metadata数据丢失，虚拟机就毁了）

11.2网络类型选择

对象存储：swift，S3

8.2 cinder控制节点的部署

安装cinder

[root@node1 ~]# yum install openstack-cinder python-cinderclient -y

 

修改cinder配置文件

[default]
rpc_backend = rabbit
glance_host = 192.168.3.199
auth_strategy = keystone

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

[oslo_messaging_rabbit]
rabbit_host = 192.168.3.199 # rabbitmq的主机
rabbit_port = 5672 # rabbitmq的端口
rabbit_userid = openstack # rabbitmq的用户
rabbit_password = openstack # rabbitmq的密码

[database]
connection = mysql://cinder:cinder@192.168.3.199/cinder # 配置mysql地址

[keystone_authtoken]
auth_uri = http://192.168.3.199:5000
auth_url = http://192.168.3.199:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder

修改后结果如下

[root@node1 cinder]# grep -n '^[a-Z]' /etc/cinder/cinder.conf 421:glance_host = 192.168.3.199536:auth_strategy = keystone2294:rpc_backend = rabbit2516:connection = mysql://cinder:cinder@192.168.3.199/cinder2641:auth_uri = http://192.168.3.199:5000 2642:auth_url = http://192.168.3.199:35357 2643:auth_plugin = password 2644:project_domain_id = default2645:user_domain_id = default2646:project_name = service2647:username = cinder2648:password = cinder2874:lock_path = /var/lib/cinder/tmp3173:rabbit_host = 192.168.3.1993177:rabbit_port = 56723189:rabbit_userid = openstack3193:rabbit_password = openstack

 

修改nova的配置文件

[root@node1 ~]# vim /etc/nova/nova.conf os_region_name = RegionOne # 通知nova使用cinder [cinder]部分

 

执行同步数据库操作

[root@node1 ~]# su -s /bin/sh -c "cinder-manage db sync" cinder检查导入数据库结果MariaDB [(none)]> use cinderDatabase changedMariaDB [cinder]> show tables;+----------------------------+| Tables_in_cinder |+----------------------------+| backups || cgsnapshots || consistencygroups || driver_initiator_data || encryption || image_volume_cache_entries || iscsi_targets || migrate_version || quality_of_service_specs || quota_classes || quota_usages || quotas || reservations || services || snapshot_metadata || snapshots || transfers || volume_admin_metadata || volume_attachment || volume_glance_metadata || volume_metadata || volume_type_extra_specs || volume_type_projects || volume_types || volumes |+----------------------------+25 rows in set (0.00 sec)

 

创建一个cinder用户,加入service项目，给予admin角色

[root@node1 ~]# source admin-openrc.sh[root@node1 ~]# openstack user create --domain default --password-prompt cinderUser Password:Repeat User Password: # （密码需要配置成cinder就是/etc/cinder/cinder.conf配置文件中配置的2648行）+-----------+----------------------------------+| Field | Value |+-----------+----------------------------------+| domain_id | default || enabled | True || id | 420d7573e9fc43b3b263f31bb6dd76e2 || name | cinder |+-----------+----------------------------------+[root@node1 ~]# openstack role add --project service --user cinder admin

 

重启nova-api服务和启动cinder服务

[root@node1 ~]# systemctl restart openstack-nova-api.service[root@node1 ~]# systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service[root@node1 ~]# systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service

创建服务（包含V1和V2）

[root@node1 ~]# openstack service create --name cinder --description "OpenStack Block Storage" volume+-------------+----------------------------------+| Field | Value |+-------------+----------------------------------+| description | OpenStack Block Storage || enabled | True || id | 6e3b2c3940d14300ab28aed272ade1d3 || name | cinder || type | volume |+-------------+----------------------------------+[root@node1 ~]# openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2+-------------+----------------------------------+| Field | Value |+-------------+----------------------------------+| description | OpenStack Block Storage || enabled | True || id | 2108489d055e4fcb8f9c88fa9d5e4e3d || name | cinderv2 || type | volumev2 |+-------------+----------------------------------+

 

分别对V1和V2创建三个环境（admin，internal，public）的endpoint

[root@node1 ~]# openstack endpoint create --region RegionOne volume public http://192.168.3.199:8776/v1/%\(tenant_id\)s+--------------+--------------------------------------------+| Field | Value |+--------------+--------------------------------------------+| enabled | True || id | 007497468db7456d81157962f8740540 || interface | public || region | RegionOne || region_id | RegionOne || service_id | 6e3b2c3940d14300ab28aed272ade1d3 || service_name | cinder || service_type | volume || url | http://192.168.3.199:8776/v1/%(tenant_id)s |+--------------+--------------------------------------------+[root@node1 ~]# openstack endpoint create --region RegionOne volume internal http://192.168.3.199:8776/v1/%\(tenant_id\)s+--------------+--------------------------------------------+| Field | Value |+--------------+--------------------------------------------+| enabled | True || id | e7543b96b69342bcabead7ad8a583860 || interface | internal || region | RegionOne || region_id | RegionOne || service_id | 6e3b2c3940d14300ab28aed272ade1d3 || service_name | cinder || service_type | volume || url | http://192.168.3.199:8776/v1/%(tenant_id)s |+--------------+--------------------------------------------+[root@node1 ~]# openstack endpoint create --region RegionOne volume admin http://192.168.3.199:8776/v1/%\(tenant_id\)s+--------------+--------------------------------------------+| Field | Value |+--------------+--------------------------------------------+| enabled | True || id | 12e4bea586384d43b16e4de5a00afb1b || interface | admin || region | RegionOne || region_id | RegionOne || service_id | 6e3b2c3940d14300ab28aed272ade1d3 || service_name | cinder || service_type | volume || url | http://192.168.3.199:8776/v1/%(tenant_id)s |+--------------+--------------------------------------------+[root@node1 ~]# openstack endpoint create --region RegionOne volumev2 public http://192.168.3.199:8776/v2/%\(tenant_id\)s+--------------+--------------------------------------------+| Field | Value |+--------------+--------------------------------------------+| enabled | True || id | 07c56b0033454fbda201f3cc58ce0a1b || interface | public || region | RegionOne || region_id | RegionOne || service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d || service_name | cinderv2 || service_type | volumev2 || url | http://192.168.3.199:8776/v2/%(tenant_id)s |+--------------+--------------------------------------------+[root@node1 ~]# openstack endpoint create --region RegionOne volumev2 internal http://192.168.3.199:8776/v2/%\(tenant_id\)s+--------------+--------------------------------------------+| Field | Value |+--------------+--------------------------------------------+| enabled | True || id | 66b34a18d4de456ab32ebba24831b959 || interface | internal || region | RegionOne || region_id | RegionOne || service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d || service_name | cinderv2 || service_type | volumev2 || url | http://192.168.3.199:8776/v2/%(tenant_id)s |+--------------+--------------------------------------------+[root@node1 ~]# openstack endpoint create --region RegionOne volumev2 admin http://192.168.3.199:8776/v2/%\(tenant_id\)s+--------------+--------------------------------------------+| Field | Value |+--------------+--------------------------------------------+| enabled | True || id | fc0bed271a7048a5aff7e63aebd0199a || interface | admin || region | RegionOne || region_id | RegionOne || service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d || service_name | cinderv2 || service_type | volumev2 || url | http://192.168.3.199:8776/v2/%(tenant_id)s |+--------------+--------------------------------------------+

 

8.3 cinder存储节点的部署（此处使用nova的计算节点即node2.chinasoft.com）

　　本文中cinder后端存储使用ISCSI（类似于nova-computer使用的kvm），ISCSI使用LVM，在定义好的VG中，每创建一个云硬盘，就会增加一个LV，使用ISCSI发布。
在存储节点上加一个硬盘

 

查看磁盘添加情况

[root@node2 ~]# fdisk -lDisk /dev/sdb: 85.9 GB, 85899345920 bytes, 167772160 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/sda: 128.8 GB, 128849018880 bytes, 251658240 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x0004c2a9Device Boot Start End Blocks Id System/dev/sda1 * 2048 616447 307200 83 Linux/dev/sda2 616448 155811839 77597696 8e Linux LVMDisk /dev/mapper/centos-root: 32.2 GB, 32212254720 bytes, 62914560 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mapper/centos-swap: 4294 MB, 4294967296 bytes, 8388608 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mapper/centos-data: 42.9 GB, 42945478656 bytes, 83877888 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytes

创建一个pv和vg（名为cinder-volumes）

[root@node2 ~]# pvcreate /dev/sdbPhysical volume "/dev/sdb" successfully created[root@node2 ~]# vgcreate cinder-volumes /dev/sdbVolume group "cinder-volumes" successfully created

修改lvm的配置文件中添加filter，只有instance可以访问

[root@node2 ~]# vim /etc/lvm/lvm.conf131 filter = [ "a/sdb/", "r/.*/"]

 

存储节点安装

[root@node2 ~]# yum install openstack-cinder targetcli python-oslo-policy -y

修改存储节点的配置文件，在这里直接拷贝控制节点的文件
[root@node1 ~]# scp /etc/cinder/cinder.conf 192.168.3.200:/etc/cinder/

修改存储(即计算节点node2.chinasoft.com)上的/etc/cinder/cinder.conf文件

添加如下配置：
默认没有[lvm]需要自己创建
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm后端存储
volume_group = cinder-volumes # vg的名称：刚才创建的
iscsi_protocol = iscsi # 使用iscsi协议
iscsi_helper = lioadm

[root@node2 cinder]# grep -n '^[a-Z]' /etc/cinder/cinder.conf
421:glance_host = 192.168.3.199
536:auth_strategy = keystone
540:enabled_backends = lvm # lvm 使用的后端是lvm，要对应添加的[lvm]，自定义也可以
2294:rpc_backend = rabbit
2516:connection = mysql://cinder:cinder@192.168.3.199/cinder
2641:auth_uri = http://192.168.3.199:5000
2642:auth_url = http://192.168.3.199:35357
2643:auth_plugin = password
2644:project_domain_id = default
2645:user_domain_id = default
2646:project_name = service
2647:username = cinder
2648:password = cinder
2874:lock_path = /var/lib/cinder/tmp
3173:rabbit_host = 192.168.3.199
3177:rabbit_port = 5672
3189:rabbit_userid = openstack
3193:rabbit_password = openstack
[lvm] # 此行不是grep过滤出来的,因为是在配置文件最后添加上的，其对应的是540行的lvm
3416:volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm后端存储
3417:volume_group = cinder-volumes # vg的名称：刚才创建的
3418:iscsi_protocol = iscsi# 使用iscsi协议
3419:iscsi_helper = lioadm

启动存储节点的cinder(这里是node2)

[root@node2 cinder]# systemctl enable openstack-cinder-volume.service target.serviceCreated symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-volume.service to /usr/lib/systemd/system/openstack-cinder-volume.service.Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.[root@node2 cinder]# systemctl start openstack-cinder-volume.service target.service

 

查看云硬盘服务状态（如果是虚拟机作为宿主机，时间不同步，会产生问题）

[root@node1 ~]# source admin-openrc.sh [root@node1 ~]# cinder service-list+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+| cinder-scheduler | node1.chinasoft.com | nova | enabled | up | 2017-04-28T09:59:54.000000 | - || cinder-volume | node2.chinasoft.com@lvm | nova | enabled | up | 2017-04-28T09:59:58.000000 | - |+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+

 

创建一个云硬盘,将云硬盘挂载到虚拟机上,启动在虚拟机实例详情可以查看到，云硬盘挂载的情况
具体步骤：
1.创建云硬盘

2.点击 动作选项下的 管理已连接云硬盘，选择需要的虚拟机 将云硬盘挂载到指定的虚拟机上

在虚拟机中对挂载的硬盘进行分区格式化,如果有时不想挂载这个云硬盘了，一定不要删掉，生产环境一定要注意，否则虚拟机会出现error，应该使用umont确定卸载了，再使用dashboard进行删除云硬盘

# ssh cirros@192.168.3.103$ sudo fdisk -lDisk /dev/vda: 1073 MB, 1073741824 bytes255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000Device Boot Start End Blocks Id System/dev/vda1 * 16065 2088449 1036192+ 83 LinuxDisk /dev/vdb: 1073 MB, 1073741824 bytes16 heads, 63 sectors/track, 2080 cylinders, total 2097152 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000Disk /dev/vdb doesn't contain a valid partition table# 分区$ sudo fdisk /dev/vdbDevice contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabelBuilding a new DOS disklabel with disk identifier 0x3fecc8a5.Changes will remain in memory only, until you decide to write them.After that, of course, the previous content won't be recoverable.Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)Command (m for help): nPartition type:p primary (0 primary, 0 extended, 4 free)e extendedSelect (default p): pPartition number (1-4, default 1): 1First sector (2048-2097151, default 2048): Using default value 2048Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): Using default value 2097151Command (m for help): wThe partition table has been altered!Calling ioctl() to re-read partition table.Syncing disks.查看分区，已经生成了新的分区/dev/vdb$ sudo fdisk -lDisk /dev/vda: 1073 MB, 1073741824 bytes255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000Device Boot Start End Blocks Id System/dev/vda1 * 16065 2088449 1036192+ 83 LinuxDisk /dev/vdb: 1073 MB, 1073741824 bytes9 heads, 8 sectors/track, 29127 cylinders, total 2097152 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0xfaacdc93Device Boot Start End Blocks Id System/dev/vdb1 2048 2097151 1047552 83 Linux# 格式化$ sudo mkfs.ext4 /dev/vdb1mke2fs 1.42.2 (27-Mar-2012)Filesystem label=OS type: LinuxBlock size=4096 (log=2)Fragment size=4096 (log=2)Stride=0 blocks, Stripe width=0 blocks65536 inodes, 261888 blocks13094 blocks (5.00%) reserved for the super userFirst data block=0Maximum filesystem blocks=2684354568 block groups32768 blocks per group, 32768 fragments per group8192 inodes per groupSuperblock backups stored on blocks: 32768, 98304, 163840, 229376Allocating group tables: done Writing inode tables: done Creating journal (4096 blocks): doneWriting superblocks and filesystem accounting information: done# 挂载$ sudo mkdir /data$ sudo mount /dev/vdb1 /data$ df -hFilesystem Size Used Available Use% Mounted on/dev 242.3M 0 242.3M 0% /dev/dev/vda1 23.2M 18.0M 4.0M 82% /tmpfs 245.8M 0 245.8M 0% /dev/shmtmpfs 200.0K 72.0K 128.0K 36% /run/dev/vdb1 1006.9M 17.3M 938.5M 2% /data

 

方法二：从云硬盘启动虚拟机

①先创建一个demo2的云硬盘

②动作-作为实例启动

九、虚拟机创建流程：

第一阶段：用户操作
1)用户使用Dashboard或者CLI连接keystone，发送用户名和密码，待keystone验证通过，keystone会返回给dashboard一个authtoken
2）Dashboard会带着上述的authtoken访问nova-api进行创建虚拟机请求
3）nova-api会通过keytoken确认dashboard的authtoken认证消息。

第二阶段：nova内组件交互阶段
4）nova-api把用户要创建的虚拟机的信息记录到数据库中.
5）nova-api使用rpc-call的方式发送请求给消息队列
6）nova-scheduler获取消息队列中的消息
7）nova-scheduler和查看数据库中要创建的虚拟机信息和计算节点的信息，进行调度
8)nova-scheduler把调度后的信息发送给消息队列
9）nova-computer获取nova-schedur发送给queue的消息
10）nova-computer通过消息队列发送消息给nova-conudctor，想要获取数据库中的要创建虚拟机信息
11）nova-conductor获取消息队列的消息
12）nova-conductor读取数据库中要创建虚拟机的信息
13）nova-conductor把从数据库获取的消息返回给消息队列
14）nova-computer获取nova-conducter返回给消息队列的信息

第三阶段：nova和其他组件进行交互
15）nova-computer通过authtoken和数据库返回的镜像id请求glance服务
16）glance会通过keystone进行认证
17）glance验证通过后把镜像返回给nova-computer
18）nova-computer通过authtoken和数据库返回的网络id请求neutron服务
19）neutron会通过keystone进行认证
20）neutron验证通过后把网络分配情况返回给nova-computer
21）nova-computer通过authtoken和数据库返回的云硬盘请求cinder服务
22）cinder会通过keystone进行认证
23）cinder验证通过后把云硬盘分配情况返回给nova-computer

第四阶段：nova创建虚拟机
24)nova-compute通过libvirt调用kvm根据已有的信息创建虚拟机，动态生成xml
25)nova-api会不断的在数据库中查询信息并在dashboard显示虚拟机的状态

生产场景注意事项：
1、新加的一个计算节点，创建虚拟机时间会很长，因为第一次使用计算节点，没有镜像，计算节点要把glance的镜像放在后端文件（/var/lib/nova/instance/_base）下，
镜像如果很大，自然会需要很长时间，然后才会在后端文件的基础上创建虚拟机（写时复制copy on write）。
2、创建虚拟机失败的原因之一：创建网桥失败。要保证eth0网卡配置文件的BOOTPROTE是static而不是dhcp状态。