自己动手写OpenStack的QoS功能
来源：互联网 发布：漏洞网站源码 编辑：程序博客网 时间：2024/06/05 16:18
http://blog.csdn.net/spch2008/article/details/9279445
本文地址：http://blog.csdn.net/spch2008/article/details/9011321
1. 创建代码ovs_qos_db.py
- '''''
- Created on 2013-6-3
- @author: spch2008
- '''
- from sqlalchemy import Column, Integer, String
- from sqlalchemy.orm import exc
- from quantum.db.models_v2 import model_base
- class QoSPortBinding(model_base.BASEV2):
- __tablename__ = 'ovs_qos_port_binding'
- port_id = Column(String(20), primary_key=True)
- qos_id = Column(String(20), primary_key=True)
- def __init__(self, port_id, qos_id):
- self.port_id = port_id
- self.qos_id = qos_id
- def __repr__(self):
- return "<QoSPortBinding(%s, %s, %s)>" % (self.port_id, self.qos_id)
将代码置于/usr/share/pyshared/quantum/plugins/openvswitch# 中
2. 引入该模块
在/usr/share/pyshared/quantum/plugins/openvswitch#下的ovs_db_v2.py中，导入上述模块
- from quantum.plugins.openvswitch import ovs_qos_db
3. 创建软连接
/usr/lib/python2.7/dist-packages/quantum/plugins/openvswitch# ln -s ../../../../../../share/pyshared/quantum/plugins/openvswitch/ovs_qos_db.py ovs_qos_db.py
4. 重启quantum-server
service quantum-server restart
本文地址：http://blog.csdn.net/spch2008/article/details/9279445
起因：OpenStack源代码看有些日子了，于是想找点东西做做。发现NVP有QoS功能，但是是针对Port的，有些复杂，
就将其简化，做一个针对Interface的QoS功能。针对Interface的QoS只需要几条命令即可，做起来方便简单。
- ovs-vsctl set interface vif0.0 ingress_policing_rate=1000
- ovs-vsctl set interface vif0.0 ingress_policing_burst=100
- ingress_policing_rate:最大发送速率（单位均为kbps）
- ingress_policing_burst:超过ingress_policing_rate的最大浮动值
过程：
1.首先通过命令创建QoS条目：
- root@nova-network:~# quantum qos-create --name spch2008 --rate 1000 --burst 100
- Created a new ovsqos:
- +-----------+--------------------------------------+
- | Field | Value |
- +-----------+--------------------------------------+
- | burst | 100 |
- | id | 41d1c686-3298-40bf-a114-b20e7c4e370c |
- | name | spch2008 |
- | rate | 1000 |
- | tenant_id | 9b320e2822da4461967232e938d92d18 |
- +-----------+--------------------------------------+
- nova boot --flavor 1 --image 4d3175cf-9d82-44de-862b-f94bd6e71fb7
- --key_name mykey instance003 --security_groups default
- --interface_qos 41d1c686-3298-40bf-a114-b20e7c4e370c
3.查看虚拟机
- root@nova-compute1:~# ovs-vsctl list Interface
- _uuid : 2542b505-c895-4e16-9dfc-535c1a922109
- admin_state : up
- cfm_fault : []
- cfm_mpid : []
- cfm_remote_mpids : []
- duplex : full
- external_ids : {attached-mac="fa:16:3e:d8:e1:7e", iface-id="d92f45e9-9c41-424d-9410-2dc5bd82aeca", iface-status=active,
- vm-uuid="0062f57a-1fda-444b-a7a5-63dafa8b87d4"}
- ingress_policing_burst: 100
- ingress_policing_rate: 1000
- lacp_current : []
- link_resets : 2
- link_speed : 10000000000
- link_state : up
- mac : []
- mtu : 1500
- name : "qvod92f45e9-9c"
- ofport : 8
- options : {}
- other_config : {}
- statistics : {collisions=0, rx_bytes=1336, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=14,
- tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
- status : {driver_name=veth, driver_version="1.0", firmware_version="N/A"}
- type : ""
思路：
2. 创建QoS-Port-Binding数据库，记录port_id 与 qos_id绑定关系。
3. 创建虚拟机时，nova调用Quantum暴露出来的API，将绑定关系写入数据库。
4. ovs-agent通过远程调用函数（参数port_id）向ovs-plugin取得QoS规则。
5. ovs-agent将规则施行于Interface上。
开发环境：
1. 采用OpenStack F版本进行开发
2. 一个控制节点nova-control， 一个网络节点nova-network，一个计算节点nova-compute
在/usr/share/pyshared/quantum/plugins/openvswitch/ 下创建新包extensions，然后将其连接到/usr/lib/python2.7/dist-packages/quantum/plugins/openvswitch下。
在extensions下创建ovsqos.py，同样需要进行软连接。具体可参见Quantum 数据库增加新表
1. 首先，处理资源属性映射，其实就是对属性进行一些限制。
- def convert_to_unsigned_int_or_none(val):
- if val is None:
- return
- try:
- val = int(val)
- if val < 0:
- raise ValueError
- except (ValueError, TypeError):
- msg = _("'%s' must be a non negative integer.") % val
- raise qexception.InvalidInput(error_message=msg)
- return val
- # Attribute Map
- RESOURCE_ATTRIBUTE_MAP = {
- 'ovsqoss': {
- 'id': {'allow_post': False, 'allow_put': False,
- 'is_visible': True,
- 'primary_key': True},
- 'name': {'allow_post': True, 'allow_put': True,
- 'is_visible': True, 'default': '',
- 'validate': {'type:string': None}},
- 'rate': {'allow_post': True, 'allow_put': True,
- 'is_visible': True, 'default': '0',
- 'convert_to': convert_to_unsigned_int_or_none},
- 'burst': {'allow_post': True, 'allow_put': True,
- 'is_visible': True, 'default': '0',
- 'convert_to': convert_to_unsigned_int_or_none},
- 'tenant_id': {'allow_post': True, 'allow_put': False,
- 'required_by_policy': True,
- 'validate': {'type:string': None},
- 'is_visible': True},
- }
- }
RESOURCE_ATTRIBUTE_MAP对属性进行制约； allow_put 是否允许用户进行修改，即update； allow_post是否通过用户传来数值进行填充，比如id，
此值为false，如果用户提交过来的数据中有id值，则出错，也就是说id由系统进行分配，不由用户指定；is_visible该属性是否对用户可见，若为false，
用户看不到该值。
2. 同理，qos是port的一个属性，对qos也进行一些限制
- OVSQOS = 'ovsqoss'
- EXTENDED_ATTRIBUTES_2_0 = {
- 'ports': {OVSQOS: {'allow_post': True,
- 'allow_put': True,
- 'is_visible': True,
- 'default': attr.ATTR_NOT_SPECIFIED}}}
用户通过quantum port-list， port-update等命令对端口进行修改的时候，ovsqos这个属性是否可以修改，对用户可见等。
3. 最重要的一个类
注意：类名必须与文件同名，且首字母大写。
- class Ovsqos(object):
- @classmethod
- def get_name(cls):
- return "ovsqos"
- @classmethod
- def get_alias(cls):
- return "ovsqos"
- @classmethod
- def get_description(cls):
- return "OVS QoS extension."
- @classmethod
- def get_namespace(cls):
- return "http://blog.csdn.net/spch2008"
- @classmethod
- def get_updated(cls):
- return "2013-06-05T10:00:00-00:00"
- @classmethod
- def get_resources(cls):
- """ Returns Ext Resources """
- exts = []
- plugin = manager.QuantumManager.get_plugin()
- resource_name = 'ovsqos'
- collection_name = resource_name.replace('_', '-') + "s"
- params = RESOURCE_ATTRIBUTE_MAP.get(resource_name + "s", dict())
- controller = base.create_resource(collection_name,
- resource_name,
- plugin, params, allow_bulk=False)
- ex = extensions.ResourceExtension(collection_name,
- controller)
- exts.append(ex)
- return exts
- def get_extended_resources(self, version):
- if version == "2.0":
- return EXTENDED_ATTRIBUTES_2_0
- else:
- return {}
- get_extended_resources取得扩展属性，然后实际上报告给了（quantum\api\v2\router.py）中的APIRouter的
__init__的ext_mgr.extend_resources("2.0", attributes.RESOURCE_ATTRIBUTE_MAP)
- get_resources 取得资源，用于后续创建路由条目
- get_alias返回一个别名，这个名字很重要。需要在ovs_quantum_plguin.py的OVSQuantumPluginV2属性_supported_extension_aliases
加入该别名
- supported_extension_aliases = ["provider", "router", "ovsqos"]
4. quantum\extensions\extensions.py文件的最末尾
def get_extensions_path() 用于取得扩展功能文件，原代码必须使用绝对路径，但是配置文档中给的是相对路径，不一致。
遂将其改为相对路径，代码如下：
- def get_extensions_path():
- #spch
- paths = ':'.join(quantum.extensions.__path__)
- #get the prefix path
- prefix = "/".join(quantum.__path__[0].split("/")[:-1])
- if cfg.CONF.api_extensions_path:
- #split the api_extensions_path by ":"
- ext_paths = cfg.CONF.api_extensions_path.split(":")
- #add prefix for each path
- for i in range(len(ext_paths)):
- ext_paths[i] = prefix + ext_paths[i]
- ext_paths.append(paths)
- paths = ":".join(ext_paths)
- return paths
5. 将扩展功能写入配置文件中，使得OpenStack系统可以加载qos功能
quantum.conf 中：
- api_extensions_path = /quantum/plugins/openvswitch/extensions
完整代码如下：
- '''''
- Created on 2013-6-5
- @author: spch2008
- '''
- from abc import abstractmethod
- from quantum.api.v2 import attributes as attr
- from quantum.api.v2 import base
- from quantum.extensions import extensions
- from quantum.common import exceptions as qexception
- from quantum import manager
- import logging
- LOG = logging.getLogger(__name__)
- def convert_to_unsigned_int_or_none(val):
- if val is None:
- return
- try:
- val = int(val)
- if val < 0:
- raise ValueError
- except (ValueError, TypeError):
- msg = _("'%s' must be a non negative integer.") % val
- raise qexception.InvalidInput(error_message=msg)
- return val
- # Attribute Map
- RESOURCE_ATTRIBUTE_MAP = {
- 'ovsqoss': {
- 'id': {'allow_post': False, 'allow_put': False,
- 'is_visible': True,
- 'primary_key': True},
- 'name': {'allow_post': True, 'allow_put': True,
- 'is_visible': True, 'default': '',
- 'validate': {'type:string': None}},
- 'rate': {'allow_post': True, 'allow_put': True,
- 'is_visible': True, 'default': '0',
- 'convert_to': convert_to_unsigned_int_or_none},
- 'burst': {'allow_post': True, 'allow_put': True,
- 'is_visible': True, 'default': '0',
- 'convert_to': convert_to_unsigned_int_or_none},
- 'tenant_id': {'allow_post': True, 'allow_put': False,
- 'required_by_policy': True,
- 'validate': {'type:string': None},
- 'is_visible': True},
- }
- }
- OVSQOS = 'ovsqoss'
- EXTENDED_ATTRIBUTES_2_0 = {
- 'ports': {OVSQOS: {'allow_post': True,
- 'allow_put': True,
- 'is_visible': True,
- 'default': attr.ATTR_NOT_SPECIFIED}}}
- class Ovsqos(object):
- @classmethod
- def get_name(cls):
- return "ovsqos"
- @classmethod
- def get_alias(cls):
- return "ovsqos"
- @classmethod
- def get_description(cls):
- return "OVS QoS extension."
- @classmethod
- def get_namespace(cls):
- return "http://blog.csdn.net/spch2008"
- @classmethod
- def get_updated(cls):
- return "2013-06-05T10:00:00-00:00"
- @classmethod
- def get_resources(cls):
- """ Returns Ext Resources """
- exts = []
- plugin = manager.QuantumManager.get_plugin()
- resource_name = 'ovsqos'
- collection_name = resource_name.replace('_', '-') + "s"
- params = RESOURCE_ATTRIBUTE_MAP.get(resource_name + "s", dict())
- controller = base.create_resource(collection_name,
- resource_name,
- plugin, params, allow_bulk=False)
- ex = extensions.ResourceExtension(collection_name,
- controller)
- exts.append(ex)
- return exts
- def get_extended_resources(self, version):
- if version == "2.0":
- return EXTENDED_ATTRIBUTES_2_0
- else:
- return {}
- class OVSPluginBase(object):
- @abstractmethod
- def create_ovsqos(self, context, ovsqos):
- pass
- @abstractmethod
- def get_ovsqoss(self, context, filters, fields):
- pass
- @abstractmethod
- def get_ovsqos(self, context, rule_id, fields):
- pass
- @abstractmethod
- def delete_ovsqos(self, context, rule_id):
- pass
- @abstractmethod
- def update_ovsqos(self, context, rule_id, ovsqos):
- pass
本文地址：http://blog.csdn.net/spch2008/article/details/9281779
创建数据库以及相应的操作：
/usr/share/pyshared/quantum/plugins/openvswitch/ 下创建ovs_qos_db.py，
然后将其连接到/usr/lib/python2.7/dist-packages/quantum/plugins/openvswitch下，可参见Quantum 数据库增加新表
ovs_qos_db.py
- import sqlalchemy as sa
- from sqlalchemy import orm
- from sqlalchemy.orm import exc
- from quantum.db import model_base
- from quantum.db import models_v2
- from quantum.openstack.common import log as logging
- from quantum.plugins.openvswitch.extensions import ovsqos
- LOG = logging.getLogger(__name__)
- class QualityOfServicePortBinding(model_base.BASEV2):
- __tablename__ = 'ovs_qos_port_bindings'
- port_id = sa.Column(sa.String(255),
- primary_key=True)
- rule_id = sa.Column(sa.String(255))
- class QualityOfServiceRule(model_base.BASEV2, models_v2.HasId, models_v2.HasTenant):
- __tablename__ = 'ovs_qos_rules'
- name = sa.Column(sa.String(255))
- rate = sa.Column(sa.Integer)
- burst = sa.Column(sa.Integer)
- class QualityOfServiceDbMixin(ovsqos.OVSPluginBase):
- __native_bulk_support = True
- def create_ovsqos(self, context, ovsqos):
- qos_rule = ovsqos['ovsqos']
- with context.session.begin(subtransactions=True):
- qos = QualityOfServiceRule(
- name=qos_rule.get('name'),
- rate=qos_rule.get('rate'),
- burst=qos_rule.get('burst'),
- tenant_id=qos_rule.get('tenant_id'))
- context.session.add(qos)
- return self._make_qos_rule_dict(qos)
- def get_ovsqoss(self, context, filters, fields):
- return self._get_collection(context, QualityOfServiceRule,
- self._make_qos_rule_dict,
- filters=filters, fields=fields)
- def get_ovsqos(self, context, rule_id, fields=None):
- return self._make_qos_rule_dict(
- self._get_ovsqos(context, rule_id), fields)
- def _get_ovsqos(self, context, rule_id):
- try:
- return self._get_by_id(context, QualityOfServiceRule, rule_id)
- except exc.NoResultFound:
- return None
- def delete_ovsqos(self, context, rule_id):
- qos = self._get_ovsqos(context, rule_id)
- with context.session.begin(subtransactions=True):
- context.session.delete(qos)
- def update_ovsqos(self, context, rule_id, ovsqos):
- rule = ovsqos['ovsqos']
- with context.session.begin(subtransactions=True):
- qos = self._get_ovsqos(context, rule_id)
- qos.update(rule)
- return self._make_qos_rule_dict(qos)
- def _make_qos_rule_dict(self, qos, fields=None):
- res = {'id': qos.id,
- 'name': qos.name,
- 'rate': qos.rate,
- 'burst': qos.burst,
- 'tenant_id':qos.tenant_id}
- return self._fields(res, fields)
数据库很简陋，只有一些基本功能。
数据库相应操作已完成，对OVS-Plugin进行修改。
在quantum\plugins\openvswitch\ovs_quantum_plugin.py中
使OVSQuantumPluginV2继承于上一篇所写的数据库操作类QuanlityOfServiceDbMixin。
- class OVSQuantumPluginV2(db_base_plugin_v2.QuantumDbPluginV2,
- ovs_qos_db.QualityOfServiceDbMixin,
- l3_db.L3_NAT_db_mixin):
在OVSQuantumPluginV2中，将相应的服务请求转到db类中去处理
- def get_ovsqoss(self, context, filters=None, fields=None):
- qoss = super(OVSQuantumPluginV2, self).get_ovsqoss(context, filters, fields)
- return qoss
- def get_ovsqos(self, context, id, fields=None):
- qos = super(OVSQuantumPluginV2, self).get_ovsqos(context, id, fields)
- return qos
- def delete_ovsqos(self, context, id):
- result = super(OVSQuantumPluginV2, self).delete_ovsqos(context, id)
- return result
- def update_ovsqos(self, context, id, ovsqos):
- result = super(OVSQuantumPluginV2, self).update_ovsqos(context, id, ovsqos)
- return result
- def create_ovsqos(self, context, ovsqos):
- qos = super(OVSQuantumPluginV2, self).create_ovsqos(context, ovsqos)
- return qos
ovs-plugin修改完成后，要改写ovs_agent，用于对openvswitch施行qos规则。
一、创建新类
在计算节点ovs_quantum_agent.py中，增加新类
- class OVSQosAgent(object):
- def __init__(self, context, plugin_rpc, int_br, agent_id):
- self.context = context
- self.plugin_rpc = plugin_rpc
- self.int_br = int_br
- self.agent_id = agent_id
- def treat_qos_added(self, device, qos_rule):
- vif_port = self.int_br.get_vif_port_by_id(device)
- if not vif_port:
- LOG.debug("spch_device_not_in")
- return
- self.int_br.set_interface_qos(vif_port.port_name, qos_rule['rate'], qos_rule['burst'])
- def apply_qos_rules(self, devices):
- if not devices:
- LOG.debug("spch_devices_is_NULL")
- for device in devices:
- try:
- qos_rule = self.plugin_rpc.get_qos_details(self.context,
- device,
- self.agent_id)
- except Exception:
- LOG.debug(_("spch_Unable to get port qos details for device_id = %s" % device))
- self.treat_qos_added(device, qos_rule)
- def refresh_qos(self, port ):
- device = port['id']
- self.apply_qos_rules(device)
1. 改写ovs_lib，增加qos功能。
quantum\agent\linux\ovs_lib.py中的OVSBridge类中，增加以下两个函数
- def set_interface_qos(self, interface, rate, burst):
- ingress_policing_rate = "ingress_policing_rate=%s" % rate
- ingress_policing_burst = "ingress_policing_burst=%s" % burst
- args = ["set", "interface", interface, ingress_policing_rate, ingress_policing_burst]
- self.run_vsctl(args)
- def clear_interface_qos(self, interface):
- ingress_policing_rate = "ingress_policing_rate=0"
- ingress_policing_burst = "ingress_policing_burst=0"
- args = ["set", "interface", interface, ingress_policing_rate, ingress_policing_burst]
- self.run_vsctl(args)
2. 增加远程调用接口，用于向ovs-plugin取得qos规则。
在quantum\agent\rpc.py中
class PluginApi(proxy.RpcProxy):
- def get_qos_details(self, context, device, agent_id):
- urn self.call(context,
- self.make_msg('get_qos_details', device=device,
- agent_id=agent_id),
- topic=self.topic)
3. 上述get_qos_details函数中，调用了控制节点的get_qos_details函数，从数据库中取得规则，返回给ovs-agent。
在控制节点的ovs_quantum_plugin.py中
class OVSRpcCallbacks(dhcp_rpc_base.DhcpRpcCallbackMixin):
- def get_qos_details(self, rpc_context, **kwargs):
- agent_id = kwargs.get('agent_id')
- device = kwargs.get('device')
- LOG.debug("spch_get_device=%s" % device)
- if device:
- binding = ovs_db_v2.get_port_qos_binding(None, device)
- if binding:
- qos = ovs_db_v2.get_port_qos(None, binding.rule_id)
- if qos:
- return {'rate':qos.rate, 'burst':qos.burst}
- else:
- return {'rate':1, 'burst':1}
二、 使用
一中，相应的准备工作一切就绪，剩下的部分只是调用OVSQosAgent相应的操作即可。
在OVSQuantumAgent中__init__中，需要持有OVSQosAgent对象。
- self.qos_agent = OVSQosAgent(self.context, self.plugin_rpc, self.int_br, self.agent_id)
nova启动虚拟机的时候会在br-int（ovs网桥）上创建一个端口，ovs-agent会定期检测，当检测到新端口创建，即会调用treat_devices_added进行处理。
所以，添加qos的操作应该treat_devices_added函数中进行。
- def treat_devices_added(self, devices):
- resync = False
- self.qos_agent.apply_qos_rules(devices)
- ………………
三、总结
当然，这些实现都很简陋，没有相应的删除操作，只是希望能够掌握开发流程。
这一篇主要介绍quantum client的编写，使完成的功能能够通过命令行进行操作。
quantumclient\quantum\v2_0中增加ovs_qos.py（不要忘记进行软连接）。
- '''''
- Created on 2013-6-3
- @author: spch2008
- '''
- import logging
- from quantumclient.quantum.v2_0 import CreateCommand
- from quantumclient.quantum.v2_0 import DeleteCommand
- from quantumclient.quantum.v2_0 import ListCommand
- from quantumclient.quantum.v2_0 import ShowCommand
- from quantumclient.quantum.v2_0 import UpdateCommand
- class CreateQoS(CreateCommand):
- """Create QoS."""
- resource = 'ovsqos'
- log = logging.getLogger(__name__ + '.CreateQos')
- def add_known_arguments(self, parser):
- parser.add_argument(
- '--name',
- dest='name', action='store',
- help='Name of qos')
- parser.add_argument(
- '--rate',
- dest='rate', action='store',
- help='qos_rate'),
- parser.add_argument(
- '--burst',
- dest='burst',
- help='qos_burst')
- def args2body(self, parsed_args):
- body = {'ovsqos': {'name': parsed_args.name,
- 'rate': parsed_args.rate,
- 'burst': parsed_args.burst}, }
- if parsed_args.tenant_id:
- body['ovsqos'].update({'tenant_id': parsed_args.tenant_id})
- return body
- class ListQoS(ListCommand):
- """List QoS."""
- resource = 'ovsqos'
- log = logging.getLogger(__name__ + '.ListQoS')
- _formatters = {}
- list_columns = ['id', 'name', 'rate', 'burst']
- class ShowQoS(ShowCommand):
- """Show QoS."""
- resource = 'ovsqos'
- log = logging.getLogger(__name__ + '.CreateQos')
- class DeleteQoS(DeleteCommand):
- """Delete QoS."""
- log = logging.getLogger(__name__ + '.DeleteQoS')
- resource = 'ovsqos'
- class UpdateQoS(UpdateCommand):
- """Update QoS."""
- log = logging.getLogger(__name__ + '.UpdateQoS')
- resource = 'ovsqos'
quantumclient\shell.py中
COMMAND_V2 变量中增加下列命令，导入上述文件中的各个类。
- 'qos-create': utils.import_class(
- 'quantumclient.quantum.v2_0.ovs_qos.CreateQoS'),
- 'qos-list':utils.import_class(
- 'quantumclient.quantum.v2_0.ovs_qos.ListQoS'),
- 'qos-show':utils.import_class(
- 'quantumclient.quantum.v2_0.ovs_qos.ShowQoS'),
- 'qos-delete':utils.import_class(
- 'quantumclient.quantum.v2_0.ovs_qos.DeleteQoS'),
- 'qos-update':utils.import_class(
- 'quantumclient.quantum.v2_0.ovs_qos.UpdateQoS'),
quantumclient\v2_0\client.py中
class Client(object):
1. 增加路径，这两个路径直接对应ovs-plugin暴露的API。
qoss_path = "/ovsqoss"
qos_path = "/ovsqoss/%s"
2. 增加相应的API处理函数
- @APIParamsCall
- def create_ovsqos(self, body=None):
- return self.post(self.qoss_path, body=body)
- @APIParamsCall
- def list_ovsqoss(self, **_params):
- return self.get(self.qoss_path, params = _params)
- @APIParamsCall
- def show_ovsqos(self, qos_id, **_params):
- return self.get(self.qos_path % (qos_id), params=_params)
- @APIParamsCall
- def delete_ovsqos(self, qos_id):
- return self.delete(self.qos_path % (qos_id))
- @APIParamsCall
- def update_ovsqos(self, qos_id, body=None):
- return self.put(self.qos_path % (qos_id), body=body)
前面已经介绍了大部分功能，只剩下启动虚拟机的时候，将qos参数interface_qos（即qos_id）传入，同时在数据库中，
关联端口与qos_id，由于这一部分不是我写的，所以不是很了解，有空的时候在仔细看看。
- 自己动手写OpenStack的QoS功能(1)
- 自己动手写OpenStack的QoS功能(3)
- 自己动手写OpenStack的QoS功能(2)
- 自己动手写OpenStack的QoS功能(4)
- 自己动手写OpenStack的QoS功能(5)
- 自己动手写OpenStack的QoS功能(6)
- 自己动手写OpenStack的QoS功能(7)
- 自己动手写OpenStack的QoS功能
- Openstack中的网络QoS功能
- Openstack中的网络QoS功能
- Openstack CPU/Disk/network QoS 功能
- 路由器的qos功能
- OpenStack QoS
- （四）Openstack学习之WSGI：自己动手写例子
- OpenStack网络QoS
- 【交换机】配置QOS的功能原理是什么？
- 自己动手写的内存管理程序
- 自己动手写的内存管理程序
- IOS学习笔记（8）UIPageViewController UIProgressView 监听和响应键盘通知
- [已解决]Android Gallery滑动太快的问题
- HNCU1330：算法3-1：八进制数
- 文件被误删后我们该怎么办？
- 注销登录后点击浏览器页面依然能后退的解决办法
- 自己动手写OpenStack的QoS功能
- HDU1035 Prime Ring Problem
- 用三张图片详解Asp.Net 全生命周期
- 判断网页在手机浏览器和PC浏览器访问！~
- C++的输出精度控制
- storm cluster
- win7 oracle32位客户端安装找不到orandce11.dll.dbl
- load data 方式导入的数据不可以用binlog日志进行恢复，因为binlog里面不产生insert sql语句。
- 判断资源存在与否如判断一个drawable的图片存在及通过id加载图片
