OpenStack VM DHCP problem with Quantum? Guideline and real case
来源:互联网 发布:偶像学校知乎 编辑:程序博客网 时间:2024/06/05 06:38
Today, OpenStack draws many eyeballs in deploying cloud-computing environments.
When using OpenStack in practical scenarios, there will be numbers of detailed evils. One notorious bug is that booted vm sometimes cannot get an IP by DHCP automatically. Many people encountered similar problems, and proposed several solutions, including restarting quantum related services. However, this may work for some special cases, while fail on the others.
So, how to find out the crime culprit for your specified problem? In this article, we will show the guideline to locate the DHCP failure reason and demonstrate with a real case.
Debug Guideline:
0) Start a DHCP request in the vm using
sudo udhcpc
or other dhcp client.
1) Does the DHCP request reach the network node?
If not, then you should use tcpdump to capture packets at the compute node’s and the network node’s network interface (at the data network). A DHCP request usually looks like
IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:82:ee:fe, length 286
if using the following commands.
tcpdump -ni eth1 port 67 and port 68
2) If the DHCP request successfully reaches the network nodes, then make sure the quantum-dhcp-agent offers reply. This can be validated through the log file (/var/log/syslog), or by tcpdump also.
The log may look like
Jun 21 10:42:31 localhost, dnsmasq-dhcp[541]: DHCPREQUEST(tap9c753e61-fc) 50.50.1.6 fa:16:3e:82:ee:fe
Jun 21 10:42:31 localhost, dnsmasq-dhcp[541]: DHCPACK(tap9c753e61-fc) 50.50.1.6 fa:16:3e:82:ee:fe 50-50-1-6
And a DHCP Reply usually looks like
IP 50.50.1.3.67 > 50.50.1.7.68: BOOTP/DHCP, Reply, length 308
If not, make sure the quantum-* services starts successfully at the network node.
service quantum-dhcp-agent status
3) Make sure the DHCP reply goes back to the compute node using tcpdump too.
4) If the DHCP reply reach the compute node, then capture at the vm’s corresponding tap-* network interface, to make sure the reply can reach vm.
If not, then try to check the quantum-plugin-openvswitch-agent services works fine at the compute node.
service quantum-plugin-openvswitch-agent status
5) Sometimes, you may need to restart the whole nodes if problem continues appear at a special machine.
A real case
I have met a weird case.
In the case, everything seems OK. The network node gets the DHCP request and gives back the offer, while the compute node successfully gets the DHCP offer. However, the vm still cannot get IP some times, while occasionally it will get one!
I look very carefully the entire process, and make sure all services are started.
Then the only suspicious component is the OpenvSwitch.
I check the of rules at the br-int (vm’s located bridge) using
ovs-ofctl dump-flows br-int
and they looks like:
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=2219.925s, table=0, n_packets=0, n_bytes=85038, idle_age=3, priority=3,in_port=1,dl_vlan=2 actions=mod_vlan_vid:1,NORMAL
cookie=0x0, duration=2231.487s, table=0, n_packets=0, n_bytes=120021, idle_age=3, priority=1 actions=NORMAL
cookie=0x0, duration=2227.341s, table=0, n_packets=0, n_bytes=16868, idle_age=5, priority=2,in_port=1 actions=drop
They look quite normal, as all the rules are generated by the quantum-plugin-openvswitch-agent service.
I also make sure the DHCP offer reach br-int with capturing packet at it’s data network interface.
tcpdump –ni int-br-eth1 port 67 or port 68
As I guess, the DHCP offer should match rule#1 (vlan mode), and send out. However, watch a while, the n_packets does not increase, which means the DHCP offer does not match the rule.
It is strange right? Why ovs does not work as expected?
Based on my years’ experience on ovs, I think there must be some HIDDEN rule destroying the processing. Then I check more details of the rules.
ovs-appctl bridge/dump-flows br-int
HAHA, some thing now is floating outside.
duration=151s, priority=180001, n_packets=0, n_bytes=0, priority=180001,arp,dl_dst=fe:86:a7:fd:c0:4f,arp_op=2,actions=NORMAL
duration=151s, priority=180003, n_packets=0, n_bytes=0, priority=180003,arp,dl_dst=00:1a:64:99:f2:72,arp_op=2,actions=NORMAL
duration=148s, priority=3, n_packets=0, n_bytes=0, priority=3,in_port=1,dl_vlan=2,actions=mod_vlan_vid:1,NORMAL
duration=151s, priority=180006, n_packets=0, n_bytes=0, priority=180006,arp,nw_src=10.0.1.197,arp_op=1,actions=NORMAL
duration=151s, priority=180004, n_packets=0, n_bytes=0, priority=180004,arp,dl_src=00:1a:64:99:f2:72,arp_op=1,actions=NORMAL
duration=151s, priority=180002, n_packets=0, n_bytes=0, priority=180002,arp,dl_src=fe:86:a7:fd:c0:4f,arp_op=1,actions=NORMAL
duration=151s, priority=15790320, n_packets=174, n_bytes=36869, priority=15790320,actions=NORMAL
duration=151s, priority=180005, n_packets=0, n_bytes=0, priority=180005,arp,nw_dst=10.0.1.197,arp_op=2,actions=NORMAL
duration=151s, priority=180008, n_packets=0, n_bytes=0, priority=180008,tcp,nw_src=10.0.1.197,tp_src=6633,actions=NORMAL
duration=151s, priority=180007, n_packets=0, n_bytes=0, priority=180007,tcp,nw_dst=10.0.1.197,tp_dst=6633,actions=NORMAL
duration=151s, priority=180000, n_packets=0, n_bytes=0, priority=180000,udp,in_port=65534,dl_src=fe:86:a7:fd:c0:4f,tp_src=68,tp_dst=67,actions=NORMAL
table_id=254, duration=165s, priority=0, n_packets=13, n_bytes=2146, priority=0,reg0=0x1,actions=controller(reason=no_match)
table_id=254, duration=165s, priority=0, n_packets=0, n_bytes=0, priority=0,reg0=0x2,drop
See that? Packets are matching the red rule, which owns a high priority and just forward the vlan packet as NORMAL!!
So where does the rule come from?
In some version of ovs, when we start ovs without any controller specified, then it may smartly works like a L2 switch, and some rules will be added automatically.
Now how to solve the problem?
We need to tell the ovs do not be that “Smart” with the commands:
ovs-vsctl set bridge br-int fail-mode=secure
At last, the problem has puzzled our team for several weeks. During solving the problem, I summarize the guideline and wish it would be a little bit helpful.
[note] First published at 2013/06/21
- OpenStack VM DHCP problem with Quantum? Guideline and real case
- OpenStack VM DHCP problem with Quantum? Guideline and real case
- Getting through OpenStack Quantum with Open vSwitch
- Practical NoSQL - Solving a Real Problem with MongoDB and Redis
- OpenStack创建2台VM之间的DHCP服务设置
- openstack Folsom+Quantum安装
- openstack quantum 网络架构
- 【OpenStack】Quantum(Grizzly) quantum agent(OVS)工作流
- update with case and in
- Real-Time Applications with Java and CORBA
- Install Openstack with Openvswitch Plugin of Quantum on rhel6.3 by RPM Way ( by quqi99 )
- 建立openstack quantum开发环境
- 建立openstack quantum开发环境 .
- 【OpenStack】Quantum(Grizzly)中的agent
- 【OpenStack】Quantum(Grizzly)中的agent .
- 建立openstack quantum开发环境
- OpenStack Grizzly Quantum Multihost 部署
- OpenStack Grizzly Quantum Multihost 部署
- 电脑的fn锁,f1-f12与功能键 互换
- 三、校园招聘面试最终结果-经验
- Linux环境,使用C语言获得文件access、modify、change的时间
- Percona,Linux,Flashcache,硬件设备的优化经验
- android service
- OpenStack VM DHCP problem with Quantum? Guideline and real case
- ssh -X 转发的配置
- 最大公约数
- IOS委托机制详解及代码
- error C2440: “static_cast”: 无法从“void (__thiscall CMyDlg::* )(WPARAM,LPARAM)”转换为“LRESULT (__thiscall
- 配置服务器出错 /usr/libexec/gconf-sanity-check-2的退出状态为256
- 手把手教你ARC——iOS/Mac开发ARC入门和使用
- OpenDaylight--源自业界的SDN控制器
- linux 引导扇区