openstack iptables太长

Log in / Register

 

Mirantis OpenStack

• Overview 
• Code 
• Bugs 
• Blueprints 
• Translations 
• Answers

[Backport][bug/1352826] applying iptables rules takes too long when large scale deployment

Bug #1399168 reported by Alexander Ignatov on 2014-12-04

6

This bug affects 1 person

AffectsStatusImportanceAssigned toMilestone​Mirantis OpenStack

Fix Committed

Critical

Alexander Ignatov

Mirantis OpenStack 6.0

​4.1.x

Won't Fix

Critical

MOS Neutron

Mirantis OpenStack 4.1.2

​5.0.x

Won't Fix

Critical

MOS Neutron

Mirantis OpenStack 5.0.3

​5.1.x

Fix Committed

Critical

Alexander Ignatov

Mirantis OpenStack 5.1.2

​6.0.x

Fix Committed

Critical

Alexander Ignatov

Mirantis OpenStack 6.0

Also affects project (?) Also affects distribution/package Nominate for series

Bug Description

This is a backport of https://bugs.launchpad.net/neutron/+bug/1352826

Original description
================

I found the time to finishing the applying iptables rules( in neutron/agent/linux/iptables_manager.py _apply_synchronized,_modify_rules) takes nearly more than half an hour( 36 minutes in my environment) when the number of active vms in cloud is more than 980.
This will lead that the time of bringing new created port up when booting an instance will take very long, and if the vif_plugging_is_fatal is true, the vif_plugging_timeout is not big enough, booting will fail.
Although optimization on _modify_rules in patch https://review.openstack.org/#/c/77549/did help shorten the cost, but still the time is not short enough (it takes 17 minutes when the number of active vms in cloud is more than 980 in my environment).
Further optimazation on _modify_rules need be done to fit the situation of Large-scale deployment.

Tags: customer-found neutron backport scale Edit Tag help

Alexander Ignatov (aignatov) on 2014-12-04

Changed in mos:status:Triaged → In Progress

OSCI Robot (oscirobot) wrote on 2014-12-04:#1

RPM package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0.mira13.git.f90d4c0.a299407

Changeset: https://review.fuel-infra.org/1237
project: openstack/neutron
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: patchset-created

Files placed on repository:
openstack-neutron-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-bigswitch-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-brocade-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-cisco-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-hyperv-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-ibm-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-linuxbridge-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-mellanox-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-metaplugin-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-metering-agent-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-midonet-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-ml2-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-nec-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-nuage-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-ofagent-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-oneconvergence-nvsd-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-opencontrail-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-openvswitch-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-plumgrid-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-ryu-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-vmware-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
openstack-neutron-vpn-agent-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm
python-neutron-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-1237/centos

OSCI Robot (oscirobot) wrote on 2014-12-04:#2

Download full text (3.3 KiB)

DEB package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0~mira12+git.f90d4c0.a299407

Changeset: https://review.fuel-infra.org/1237
project: openstack/neutron
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: patchset-created

Files placed on repository:
neutron-common_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-dhcp-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-l3-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-lbaas-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-metadata-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-metering-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-bigswitch-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-bigswitch_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-brocade_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-cisco_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-hyperv_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-ibm-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-ibm_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-linuxbridge-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-linuxbridge_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-metaplugin_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-metering-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-midonet_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-ml2_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-mlnx-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-mlnx_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-nec-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-nec_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-nicira_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-oneconvergence-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-oneconvergence_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-openflow-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-openvswitch-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-openvswitch_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-plumgrid_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-ryu-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-ryu_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-vmware_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-plugin-vpn-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-server_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
neutron-vpn-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb
python-neutron_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb

NOTE: Changeset is not merg...

Read more...

Alexander Ignatov (aignatov) on 2014-12-04

tags:added: scaleChanged in mos:importance:High → Critical

Nastya Urlapova (aurlapova) wrote on 2014-12-05:#3

Hm, did you make the same loading as for 5.1.1, so 6.0 - more than 980 vms? Let me remind you that we certified 5.1.1 only on 20 nodes. Aleksander, please clarify this moment for 5.1.1 version.

Alexander Ignatov (aignatov) wrote on 2014-12-06:#4

Nastya, this is original description from the bug filed in upstream.

The RCA of this bug is algorithm for modifying iptables rules on each compute. So the main factor of this issue is a number of VMs per compute and number of security groups rules.

If we read comments from upstream issue we can find that bug submitter talks about 100 VMs per compute approximately:
"We have got 11 hosts, one is the controller, other 10 are compute nodes." (c)

So I did several checks to verify the patch:

TestCase1 (measuring a time of updating iptables rules):

Prerequisites:
- ~500 rules in security group "test-sg" to attach for VMs
- already deployed 96 VMs on the same compute using an availability-zone which contains the same compute for each VM
- flavor is not important here, cirros image

1. When all pre-deployed VMs are Active check that openvswich-agent doesn't consume 100% CPU
2. Start booting 1 new VM with "test-sg" security-group and start measuring time when ovs-agent consumes 100% CPU

Expected result: 100% CPU usage for ~10 sec, VM is booted success
Note: before this fix you could check that VM is being booted too long (8-10 mins and could run into Error state) and CPU about 100% for the whole period of booting

Second check is simple:

1. Deploy 10 VMs on dedicated compute, attach "test-sg" for each instance.
2. Deploy 20 VMs on dedicated compute, attach "test-sg" for each instance.
3 Deploy 50 VMs on dedicated compute, attach "test-sg" for each instance.

Expected result: all 80 VMs are up and runnig
Note: before fix number of failed VMs was about 90%-30%

All my tests are passed I assume that this bug is fixed correctly and proposing to merge it.

Alexander Ignatov (aignatov) wrote on 2014-12-06:#5

Also I've performed the following check using WARNING logs in python at the beginning and ending of function which modifies security group rules using "START MODIFY RULES" and "END MODIFY RULES" labels:

After patch:

There was booted 96 VMs and 460 rules per VM

Adding +1 VM
2014-12-05 22:51:46.616 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] START MODIFY RULES
2014-12-05 22:51:57.647 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] END MODIFY RULES

Result: 11 sec

Adding +1 VM
2014-12-05 22:55:35.525 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] START MODIFY RULES
2014-12-05 22:55:46.934 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] END MODIFY RULES
Result: 11 sec

After deleting 4VMs
2014-12-05 23:01:21.946 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] START MODIFY RULES
2014-12-05 23:01:33.097 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] END MODIFY RULES
Result: 11 sec

Before patch

There was booted 96 VMs and 460 rules per VM

2014-12-05 23:28:30.195 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] START MODIFY RULES
2014-12-05 23:38:38.574 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] END MODIFY RULES
Result: 10 mins

Booted 1 VM and
2014-12-05 23:47:12.682 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] START MODIFY RULES
2014-12-05 23:57:13.310 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] END MODIFY RULES

Result: operation of modifying iptables took 10 min and VM moved into Error state

Deleting of 20 VMs
2014-12-06 00:00:18.216 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] START MODIFY RULES
2014-12-06 00:10:04.423 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] END MODIFY RULES

Result: operation of modifying iptables took 10 min

Alexander Ignatov (aignatov) wrote on 2014-12-06:#6

I think patch is ready to be merged

OSCI Robot (oscirobot) wrote on 2014-12-06:#7

RPM package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0.mira13

Changeset: https://review.fuel-infra.org/1237
project: openstack/neutron
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: change-merged

Files placed on repository:
openstack-neutron-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-bigswitch-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-brocade-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-cisco-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-hyperv-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-ibm-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-linuxbridge-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-mellanox-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-metaplugin-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-metering-agent-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-midonet-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-ml2-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-nec-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-nuage-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-ofagent-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-oneconvergence-nvsd-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-opencontrail-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-openvswitch-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-plumgrid-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-ryu-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-vmware-2014.2-fuel6.0.mira13.noarch.rpm
openstack-neutron-vpn-agent-2014.2-fuel6.0.mira13.noarch.rpm
python-neutron-2014.2-fuel6.0.mira13.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos

OSCI Robot (oscirobot) wrote on 2014-12-06:#8

DEB package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0~mira12

Changeset: https://review.fuel-infra.org/1237
project: openstack/neutron
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: change-merged

Files placed on repository:
neutron-common_2014.2-fuel6.0~mira12_all.deb
neutron-dhcp-agent_2014.2-fuel6.0~mira12_all.deb
neutron-l3-agent_2014.2-fuel6.0~mira12_all.deb
neutron-lbaas-agent_2014.2-fuel6.0~mira12_all.deb
neutron-metadata-agent_2014.2-fuel6.0~mira12_all.deb
neutron-metering-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-bigswitch-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-bigswitch_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-brocade_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-cisco_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-hyperv_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-ibm-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-ibm_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-linuxbridge-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-linuxbridge_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-metaplugin_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-metering-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-midonet_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-ml2_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-mlnx-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-mlnx_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-nec-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-nec_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-nicira_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-oneconvergence-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-oneconvergence_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-openflow-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-openvswitch-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-openvswitch_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-plumgrid_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-ryu-agent_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-ryu_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-vmware_2014.2-fuel6.0~mira12_all.deb
neutron-plugin-vpn-agent_2014.2-fuel6.0~mira12_all.deb
neutron-server_2014.2-fuel6.0~mira12_all.deb
neutron-vpn-agent_2014.2-fuel6.0~mira12_all.deb
python-neutron_2014.2-fuel6.0~mira12_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable/ubuntu

Mike Scherbakov (mihgen) wrote on 2014-12-07:#9

Colleagues, why is it Incomplete for 5.1.2? I thought it affects 5.1.X. Let's figure out: if it does not, then the status of this bug should be Invalid for 5.1.1 / 5.1.2.

OSCI Robot (oscirobot) wrote on 2014-12-11:#10

RPM package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2.mira2.git.b0dd5db.4d22b5f

Changeset: https://review.fuel-infra.org/1319
project: openstack/neutron
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: patchset-created

Files placed on repository:
openstack-neutron-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-bigswitch-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-brocade-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-cisco-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-hyperv-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-ibm-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-linuxbridge-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-mellanox-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-metaplugin-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-metering-agent-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-midonet-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-ml2-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-nec-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-nuage-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-ofagent-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-oneconvergence-nvsd-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-openvswitch-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-plumgrid-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-ryu-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-vmware-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
openstack-neutron-vpn-agent-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm
python-neutron-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-1319/centos

OSCI Robot (oscirobot) wrote on 2014-12-11:#11

Download full text (3.4 KiB)

DEB package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2~mira2+git.b0dd5db.4d22b5f

Changeset: https://review.fuel-infra.org/1319
project: openstack/neutron
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: patchset-created

Files placed on repository:
neutron-common_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-dhcp-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-l3-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-lbaas-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-metadata-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-metering-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-bigswitch-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-bigswitch_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-brocade_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-cisco_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-hyperv_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-ibm-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-ibm_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-linuxbridge-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-linuxbridge_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-metaplugin_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-metering-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-midonet_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-ml2_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-mlnx-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-mlnx_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-nec-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-nec_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-nicira_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-oneconvergence-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-oneconvergence_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-openflow-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-openvswitch-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-openvswitch_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-plumgrid_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-ryu-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-ryu_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-vmware_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-plugin-vpn-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-server_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb
neutron-vpn-agent_2014.1.3-fuel5.1.2~mira2+git.b0...

Read more...

OSCI Robot (oscirobot) wrote on 2014-12-15:#12

RPM package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2.mira2

Changeset: https://review.fuel-infra.org/1319
project: openstack/neutron
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: change-merged

Files placed on repository:
openstack-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-bigswitch-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-brocade-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-cisco-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-hyperv-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-ibm-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-linuxbridge-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-mellanox-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-metaplugin-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-metering-agent-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-midonet-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-ml2-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-nec-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-nuage-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-ofagent-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-oneconvergence-nvsd-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-openvswitch-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-plumgrid-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-ryu-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-vmware-2014.1.3-fuel5.1.2.mira2.noarch.rpm
openstack-neutron-vpn-agent-2014.1.3-fuel5.1.2.mira2.noarch.rpm
python-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable/centos

OSCI Robot (oscirobot) wrote on 2014-12-15:#13

DEB package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2~mira2

Changeset: https://review.fuel-infra.org/1319
project: openstack/neutron
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: change-merged

Files placed on repository:
neutron-common_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-dhcp-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-l3-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-lbaas-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-metadata-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-metering-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-bigswitch-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-bigswitch_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-brocade_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-cisco_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-hyperv_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-ibm-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-ibm_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-linuxbridge-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-linuxbridge_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-metaplugin_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-metering-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-midonet_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-ml2_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-mlnx-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-mlnx_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-nec-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-nec_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-nicira_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-oneconvergence-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-oneconvergence_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-openflow-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-openvswitch-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-openvswitch_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-plumgrid_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-ryu-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-ryu_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-vmware_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-plugin-vpn-agent_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-server_2014.1.3-fuel5.1.2~mira2_all.deb
neutron-vpn-agent_2014.1.3-fuel5.1.2~mira2_all.deb
python-neutron_2014.1.3-fuel5.1.2~mira2_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable/ubuntu

Alexander Ignatov (aignatov) wrote on 2014-12-22:#16

Release note for 5.1.1:

ISSUE: Virtual Machines fail to start due to slow processing of IP tables rules

SUMMARY: In some larger scale environments, virtual machines fail to launch due to slow processing of IP tables rules. Environments with large numbers of compute nodes and VMs can lead to large IP tables in total if they apllied. Slow processing of large IP tables can lead to service timeouts that may prevent virtual machines from launching.

The problem is fixed in python-neutron package from MOS 5.1.2 (attached to the LP ticket).

Steps to apply patches on CentOS environments:
(should be done one by one on all compute nodes, then on all controllers, also one by one)

Upload package python-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpm to current node:
Execute the following on the node:
Run the following commands:
[root@node-2 ~]# rpm -Uvh --nodeps python-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpm
warning: python-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID bf906159: NOKEY
Preparing... ########################################### [100%]
 1:python-neutron ########################################### [100%]
Make sure step 2 doesn’t produce any errors
On computes nodes do: service neutron-openvswitch-agent restart
On one of controller nodes: crm resource restart clone_p_neutron-openvswitch-agent

Dmitry Borodaenko (dborodaenko) wrote on 2014-12-29:#18

If it was a Neutron bug, how is it possible that this had to be fixed in 5.1.x but doesn't reproduce in 5.0.x?

Dmitry Borodaenko (dborodaenko) wrote on 2014-12-29:#19

We also still need a backport of this for 4.1.x, status changed from Won't Fix to Confirmed.

Dmitry Borodaenko (dborodaenko) wrote on 2015-01-26:#20

No updates since December, what's the status of the backports?

Alexander Ignatov (aignatov) wrote on 2015-01-29:#21

For this patch which affects 4.x and 5.x versions Mirantis published official Tech Bulletin:

https://online.mirantis.com/hs-fs/hub/197500/file-2386174558-pdf/pdf/Mirantis-Technical-Bulletin-2014121702-Slow-Processing-of-IP-Tables.pdf

Consider to move this bug to Won't fix.

See full activity log

To post a comment you must log in.

• Report a bug

This report contains Public information  Edit

Everyone can see this information.

• Mark as duplicate
• Convert to a question
• Link a related branch
• Link to CVE

You are not directly subscribed to this bug's notifications.

• Edit bug mail

Other bug subscribers

Subscribe someone else

Notified of all changes

• Alexander Ignatov

May be notified

• Dmitry Burmistrov
• Dmitry Mescheryakov
• MOS Neutron
• Michael Rowland H...
• Ruslan Kamaldinov
• Vasilios Tzanoudakis
• Yaroslav Lobankov

  •  Take the tour  •  Read the guide   

 

© 2004-2015 Canonical Ltd.  •  Terms of use  •  Contact Launchpad Support  •  Blog  •  Careers  •  System status  •  r17413 (Get the code!)