Test multipath feature by openstack (by quqi99)

**作者：张华 发表于：2017-02-10
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
( http://blog.csdn.net/quqi99 )**

juju快速搭建openstack

附件中的basic.yaml文件对于cinder的配置是将cinder-api, cinder-schedule, cinder-volume三个服务安装一个cinder/0节点中

#bzr branch lp:openstack-charm-testingjuju destroy-environment --force zhhuabjjuju switch zhhuabj && juju bootstrap juju-deployer -c ./basic.yaml -d xenial-mitakasource  ~/openstack-charm-testing/novarccd ~/openstack-charm-testing./configure

存储节点硬件配置

1, /etc/ceph/ceph.conf中的volume_group选项默认为cinder-volumes，故需先创建卷组

dd if=/dev/zero of=/images/cinder-volumes.img bs=1M count=4096 oflag=directsgdisk -g --clear /images/cinder-volumes.imgsudo vgcreate cinder-volumes $(sudo losetup --show -f /images/cinder-volumes.img)#sudo pvcreate /dev/vdd#sudo vgcreate cinder-volumes /dev/vdd#sudo lvcreate -L2G -nceph0 cinder-volumes$ sudo vgs  VG             #PV #LV #SN Attr   VSize VFree  cinder-volumes   1   1   0 wz--n- 4.00g 3.00g

2, multipath要求存储节点有两块网卡，故需为存储节点再添加一块网卡

source ~/novarcnova interface-attach <cinder_node_uuid> --net-id=<zhhuabj_admin_net_uuid>source ~/openstack-charm-testing/novarcjuju ssh cinder/0 sudo ifconfig ens7 upjuju ssh cinder/0 sudo dhclient ens7$ juju ssh cinder/0 sudo ip addr show ens3 |grep global    inet 10.5.9.15/16 brd 10.5.255.255 scope global ens3$ juju ssh cinder/0 sudo ip addr show ens7 |grep global    inet 10.5.9.27/16 brd 10.5.255.255 scope global ens7

创建虚机

nova keypair-add --pub-key ~/.ssh/id_rsa.pub mykeynova secgroup-add-rule default tcp 22 22 0.0.0.0/0nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0nova boot --key-name mykey --image trusty --flavor m1.small --nic net-id=$(neutron port-list |grep ' private ' |awk '{print $2}') i1FLOATING_IP=$(nova floating-ip-create |grep 'ext_net' |awk '{print $4}')nova add-floating-ip i1 $FLOATING_IPssh ubuntu@10.5.150.1 -v

openstack端的multipath配置

1, /etc/nova/nova.conf

[libvirt]iscsi_use_multipath = True

2, /etc/cinder/cinder.conf

[default]iscsi_ip_address=10.5.9.15iscsi_secondary_ip_addresses = 10.5.9.27

3, 配置完后，重启ceph-volume服务

sudo service cinder-volume restart

创建卷

cinder create --display_name test_volume 1

cinder也支持下面方式创建multi-backend, 不过本例未用到。

[lvmdriver-1]volume_group = stack-volumes-lvmdriver-1volume_driver = cinder.volume.drivers.lvm.LVMISCSIDrivervolume_backend_name = lvmbackend[lvmdriver-2]volume_group = stack-volumes-lvmdriver-2volume_driver = cinder.volume.drivers.lvm.LVMISCSIDrivervolume_backend_name = lvmbackendcinder type-create type-testcinder type-key type-test set volume_backend_name=lvmbackendcinder service-listcinder create --display_name test_volume --volume_type type-test 1

attach卷

#nova volume-detach i1 3aa1d2c9-2eba-4b44-b95d-79d8b11f7246nova volume-attach i1 3aa1d2c9-2eba-4b44-b95d-79d8b11f7246

attach卷到虚机之后，配置iscsi_ip_address=10.5.9.15与iscsi_secondary_ip_addresses = 10.5.9.27将会被更新到provider_location表中.

mysql -ucinder -p6HxgTxcnrh7hpBTzChxfSN8FnLVXdxHh -h10.5.9.19mysql> select provider_location from volumes where id='3aa1d2c9-2eba-4b44-b95d-79d8b11f7246';+---------------------------------------------------------------------------------------------------------+| provider_location                                                                                       |+---------------------------------------------------------------------------------------------------------+| 10.5.9.15:3260;10.5.9.27:3260,1 iqn.2010-10.org.openstack:volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246 1 |+---------------------------------------------------------------------------------------------------------

nova端connection_info表也会更新

mysql -unova -pWhhRjy7C2mqBhm46FZPxpyNY7mWbKLtT -h10.5.9.19select connection_info from block_device_mapping where instance_uuid='f6319709-5236-441c-b420-d63f3f4b0382'; 

这一块的完整代码流程分析如下:

1, Create volume (vol_id) in cinder side, the cinder option iscsi_ip_address will be updated to DB table volume.2, Using vol_id to boot a VM.nova boot --image <image_id> --flavor 2 --key-name mykey --block-device-mapping vda=<vol_id>:<type>:<size>:<delete-on-terminate> <instance_name>The block_device_mapping data stucture looks like:{'block_device_mapping': [{'connection_info': {u'driver_volume_type': u'iscsi','serial': u'b66e294e-b997-48c1-9208-817be475e95b',u'data': {u'target_discovered': False,u'target_iqn': u'iqn.2010-10.org.openstack: volume-b66e294e-b997-48c1-9208-817be475e95b',u'target_portal': u'192.168.82.231: 3260',u'volume_id': u'b66e294e-b997-48c1-9208-817be475e95b',u'target_lun': 1,u'auth_password': u'jcYpzNiA4ZQ4dyiC26fB',u'auth_username': u'CQZto4sC4HKkx57U4WfX',u'auth_method': u'CHAP'}},'mount_device': u'vda','delete_on_termination': False}],'root_device_name': None,'ephemerals': [],'swap': None}3, nova gets above connection_info from cinder by invoking initialize_connection() only when attaching a volume to a VM.def attach(self, context, instance, volume_api, virt_driver,do_check_attach=True, do_driver_attach=False):...connection_info = volume_api.initialize_connection(context, self.volume_id, connector)4, initialize_connection() in cinder querys volume from DB table volume by volume_id,def initialize_connection(self, context, volume_id, connector):...volume = self.db.volume_get(context, volume_id)model_update = Nonetry:LOG.debug(_("Volume %s: creating export"), volume_id)model_update = self.driver.create_export(context.elevated(), volume)if model_update:volume = self.db.volume_update(context,volume_id,model_update)def create_export(self, context, volume, volume_path):iscsi_name = "%s%s" % (CONF.iscsi_target_prefix, volume['name'])iscsi_target, lun = self._get_target_and_lun(context, volume)...tid = self.create_iscsi_target(iscsi_name, iscsi_target, 0, volume_path, chap_auth)data = {}data['location'] = self._iscsi_location(CONF.iscsi_ip_address, tid, iscsi_name, lun)return data5, _get_iscsi_properties will implement the transform between provider_location and target_portaldef _get_iscsi_properties(self, volume):properties = {}location = volume['provider_location']if location:# provider_location is the same format as iSCSI discovery outputproperties['target_discovered'] = Falseelse:location = self._do_iscsi_discovery(volume)properties['target_discovered'] = Trueresults = location.split(" ")properties['target_portal'] = results[0].split(",")[0]

一些有用输出

1, cinder端

$ sudo tgtadm --mode target --op showTarget 1: iqn.2010-10.org.openstack:volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246    System information:        Driver: iscsi        State: ready    I_T nexus information:        I_T nexus: 5            Initiator: iqn.1993-08.org.debian:01:bc814ff1c89 alias: juju-zhhuabj-machine-9            Connection: 0                IP Address: 10.5.9.24        I_T nexus: 6            Initiator: iqn.1993-08.org.debian:01:bc814ff1c89 alias: juju-zhhuabj-machine-9            Connection: 0                IP Address: 10.5.9.24    LUN information:        LUN: 0            Type: controller            SCSI ID: IET     00010000            SCSI SN: beaf10            Size: 0 MB, Block size: 1            Online: Yes            Removable media: No            Prevent removal: No            Readonly: No            SWP: No            Thin-provisioning: No            Backing store type: null            Backing store path: None            Backing store flags:         LUN: 1            Type: disk            SCSI ID: IET     00010001            SCSI SN: beaf11            Size: 1074 MB, Block size: 512            Online: Yes            Removable media: No            Prevent removal: No            Readonly: No            SWP: No            Thin-provisioning: No            Backing store type: rdwr            Backing store path: /dev/cinder-volumes/volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246            Backing store flags:     Account information:        e3X9c6oh3tSLB6pyDjpn    ACL information:        ALL

cinder做得这些相当于下列手工命令做的事情：

$ sudo apt-get install tgt$ sudo tgtadm --lld iscsi --op new --mode target --tid 1 --targetname iqn.2010-10.org.openstack:volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246#Attach a logical unit ( LUN )$ sudo tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 --backing-store /dev/cinder-volumes/volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246# cat /etc/tgt/targets.conf include /etc/tgt/conf.d/*.conf# cat /etc/tgt/conf.d/cinder_tgt.confinclude /var/lib/cinder/volumes/*# cat /var/lib/cinder/volumes/volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246 <target iqn.2010-10.org.openstack:volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246>    backing-store /dev/cinder-volumes/volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246    driver iscsi    incominguser e3X9c6oh3tSLB6pyDjpn 6w49MzSf7iPk9g3H    write-cache on</target>$ sudo iscsiadm -m discovery -t sendtargets -p 10.5.1.30$ sudo iscsiadm -m discovery -t sendtargets -p 10.5.1.31$ sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246 -p 10.5.1.30 --login$ sudo iscsiadm -m node -T iqn.2010-10.org.openstack:volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246 -p 10.5.1.31 --login

2, nova端：

$ sudo multipath -ll360000000000000000e00000000010001 dm-0 IET,VIRTUAL-DISKsize=1.0G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 10:0:0:1 sdb 8:16 active ready running`-+- policy='round-robin 0' prio=1 status=enabled  `- 9:0:0:1  sda 8:0  active ready running$ sudo iscsiadm -m node10.5.9.15:3260,-1 iqn.2010-10.org.openstack:volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f724610.5.9.27:3260,-1 iqn.2010-10.org.openstack:volume-3aa1d2c9-2eba-4b44-b95d-79d8b11f7246

附录basic.yaml

base-services:  services:    rabbitmq-server:      branch: https://github.com/openstack/charm-rabbitmq-server      constraints: mem=1G    keystone:      branch: https://github.com/openstack/charm-keystone      constraints: mem=1G      options:        admin-password: openstack        admin-token: ubuntutesting    nova-compute:      branch: https://github.com/openstack/charm-nova-compute      num_units: 2      constraints: mem=4G      options:        enable-live-migration: False        enable-resize: True        #migration-auth-type: ssh    nova-cloud-controller:      branch: https://github.com/openstack/charm-nova-cloud-controller      constraints: mem=1G      options:        network-manager: Neutron    neutron-gateway:      branch: https://github.com/openstack/charm-neutron-gateway      constraints: mem=2G      options:        instance-mtu: 1300        bridge-mappings: physnet1:br-ex    neutron-api:      branch: https://github.com/openstack/charm-neutron-api      constraints: mem=1G      options:        neutron-security-groups: True        flat-network-providers: physnet1    neutron-openvswitch:      branch: https://github.com/openstack/charm-neutron-openvswitch    cinder:      branch: https://github.com/openstack/charm-cinder      options:        block-device: "sdc"        glance-api-version: 2      constraints: mem=1G    glance:      branch: https://github.com/openstack/charm-glance      constraints: mem=1G    mongodb:      branch: lp:charms/trusty/mongodb      constraints: mem=1G  relations:    - [ keystone, mysql ]    - [ nova-cloud-controller, mysql ]    - [ nova-cloud-controller, rabbitmq-server ]    - [ nova-cloud-controller, glance ]    - [ nova-cloud-controller, keystone ]    - [ nova-compute, nova-cloud-controller ]    - [ nova-compute, mysql ]    - - nova-compute      - rabbitmq-server:amqp    - [ nova-compute, glance ]    - [ glance, mysql ]    - [ glance, keystone ]    - [ glance, "cinder:image-service" ]    - [ glance, rabbitmq-server ]    - [ cinder, mysql ]    - [ cinder, rabbitmq-server ]    - [ cinder, nova-cloud-controller ]    - [ cinder, keystone ]    - [ neutron-gateway, nova-cloud-controller ]    - [ "neutron-gateway:amqp", rabbitmq-server ]    - [ neutron-api, mysql ]    - [ neutron-api, rabbitmq-server ]    - [ neutron-api, nova-cloud-controller ]    - [ neutron-api, neutron-openvswitch ]    - [ neutron-api, keystone ]    - [ neutron-api, neutron-gateway ]    - [ neutron-openvswitch, nova-compute ]    - [ neutron-openvswitch, rabbitmq-server ]openstack-services-legacy:  inherits: base-services  services:    mysql:      branch: lp:charms/trusty/mysql      constraints: mem=2G      options:        dataset-size: 50%        max-connections: 20000openstack-services:  inherits: base-services  services:    mysql:      branch: https://github.com/openstack/charm-percona-cluster      constraints: mem=4G      options:        dataset-size: 50%        max-connections: 20000        root-password: ChangeMe123        sst-password: ChangeMe123# icehouseprecise-icehouse:  inherits: openstack-services-legacy  series: precise  overrides:    openstack-origin: cloud:precise-icehouse    source: cloud:precise-icehouseprecise-icehouse-proposed:  inherits: precise-icehouse  overrides:    openstack-origin: cloud:precise-icehouse/proposed    source: cloud:precise-updates/icehouseprecise-icehouse-staging:  inherits: precise-icehouse  overrides:    openstack-origin: ppa:ubuntu-cloud-archive/icehouse-staging    source: ppa:ubuntu-cloud-archive/icehouse-stagingprecise-icehouse-trunk:  inherits: precise-icehouse  overrides:    openstack-origin: ppa:openstack-ubuntu-testing/icehouse    source: ppa:openstack-ubuntu-testing/icehousetrusty-icehouse:  inherits: openstack-services  series: trustytrusty-icehouse-proposed:  inherits: trusty-icehouse  overrides:    source: proposed    openstack-origin: distro-proposedtrusty-icehouse-trunk:  inherits: trusty-icehouse  overrides:    openstack-origin: ppa:openstack-ubuntu-testing/icehouse    source: ppa:openstack-ubuntu-testing/icehouse    offline-compression: "no"# kilotrusty-kilo:  inherits: openstack-services  series: trusty  overrides:    openstack-origin: cloud:trusty-kilo    source: cloud:trusty-kilotrusty-kilo-proposed:  inherits: trusty-kilo  overrides:    openstack-origin: cloud:trusty-kilo/proposed    source: cloud:trusty-kilo/proposedtrusty-kilo-staging:  inherits: trusty-kilo  overrides:    openstack-origin: ppa:ubuntu-cloud-archive/kilo-staging    source: ppa:ubuntu-cloud-archive/kilo-staging# libertytrusty-liberty:  inherits: openstack-services  series: trusty  overrides:    openstack-origin: cloud:trusty-liberty    source: cloud:trusty-libertytrusty-liberty-proposed:  inherits: trusty-liberty  overrides:    openstack-origin: cloud:trusty-liberty/proposed    source: cloud:trusty-liberty/proposedtrusty-liberty-staging:  inherits: trusty-liberty  overrides:    openstack-origin: ppa:ubuntu-cloud-archive/liberty-staging    source: ppa:ubuntu-cloud-archive/liberty-staging# libertywily-liberty:  inherits: openstack-services  series: wilywily-liberty-proposed:  inherits: wily-liberty  overrides:    source: proposed    openstack-origin: distro-proposed# mitakatrusty-mitaka:  inherits: openstack-services  series: trusty  overrides:    openstack-origin: cloud:trusty-mitaka    source: cloud:trusty-mitakatrusty-mitaka-proposed:  inherits: trusty-mitaka  overrides:    openstack-origin: cloud:trusty-mitaka/proposed    source: cloud:trusty-mitaka/proposedtrusty-mitaka-staging:  inherits: trusty-mitaka  overrides:    openstack-origin: ppa:ubuntu-cloud-archive/mitaka-staging    source: ppa:ubuntu-cloud-archive/mitaka-stagingxenial-mitaka:  inherits: openstack-services  series: xenialxenial-mitaka-proposed:  inherits: xenial-mitaka  overrides:    source: proposed    openstack-origin: distro-proposed# newtonxenial-newton:  inherits: openstack-services  series: xenial  overrides:    openstack-origin: cloud:xenial-newton    source: cloud:xenial-newtonxenial-newton-proposed:  inherits: xenial-newton  overrides:    openstack-origin: cloud:xenial-newton/proposed    source: cloud:xenial-newton/proposedxenial-newton-staging:  inherits: xenial-newton  overrides:    openstack-origin: ppa:ubuntu-cloud-archive/newton-staging    source: ppa:ubuntu-cloud-archive/newton-stagingxenial-newton-branch:  inherits: xenial-newton  overrides:    openstack-origin: ppa:openstack-ubuntu-testing/newton    source: ppa:openstack-ubuntu-testing/newtonyakkety-newton:  inherits: openstack-services  series: yakketyyakkety-newton-proposed:  inherits: yakkety-newton  overrides:    source: proposed    openstack-origin: distro-proposedyakkety-newton-branch:  inherits: yakkety-newton  overrides:    openstack-origin: ppa:openstack-ubuntu-testing/newton    source: ppa:openstack-ubuntu-testing/newton