MessagingTimeout: Timed out waiting for a reply to message ID
来源:互联网 发布:折800打开无网络 编辑:程序博客网 时间:2024/04/27 13:47
l3中出现大量消息超时错误,对网络的操作各种异常。
报错如下:
2016-02-25 05:54:59.886 15110 ERROR neutron.agent.l3.agent [req-db9207e6-9270-4f23-8c19-0d91d20cc6fb ] Failed synchronizing routers due to RPC error
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 593, in fetch_and_sync_all_routers
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent routers = self.plugin_rpc.get_routers(context)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 93, in get_routers
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent router_ids=router_ids)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent retry=self.retry)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent timeout=timeout, retry=retry)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent retry=retry)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent result = self._waiter.wait(msg_id, timeout)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent message = self.waiters.get(msg_id, timeout=timeout)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent 'to message ID %s' % msg_id)
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent MessagingTimeout: Timed out waiting for a reply to message ID d4baae114cee4f6d831c5eec3c5f0de3
2016-02-25 05:54:59.886 15110 TRACE neutron.agent.l3.agent
所有超时都指向同步路由的操作。
而且同步失败时,rabbit中的队列q-l3-plugin中有大量未应答消息积压,该队列为同步路由时使用,路由同步时会使用消息队列传送所有路由的属性详情,消息量很大
1)测试是否由于消息太大导致,编写测试代码,尝试连续1000次发送该消息,并未出现丢失消息的情况,
2)尝试减少路由器数量,短时内情况有所改善,但是随时间增加,消息积压依然有更加严重的趋势
3)尝试合入K版本oslo_messaging的最新更新,未有改善
最终跟踪neutron代码,发现消息队列出现Timeout的原因是:
neutron在同步路由信息时,会从neutron-server获取所有router的信息,这个过程会比较长(130s左右,和网络资源的多少有关系),而 在/etc/neutron/neutron.conf中会有一个配置项“rpc_response_timeout”,它用来配置RPC的超时时间,默认为60s,所以导致超时异常.解决方法为设置rpc_response_timeout=180.
延时是解决各种问题的大招啊。。。
- MessagingTimeout: Timed out waiting for a reply to message ID
- DHCP request error:Timed out waiting for dhcpcd to start
- [errorCode=3 message=Timed out waiting to receive cluster topology
- Timed out waiting for signal from JVM.
- Memcached Timed out(1000) waiting for operation
- Timed out waiting for signal from JVM. .
- timed out waiting for device to boot,unable to boot the iOS Simulator
- com.mchange.v2.resourcepool.TimeoutException: A client timed out while waiting to acquire a resource
- Oracle RAC root.sh 报错 Timed out waiting for the CRS stack to start 解决方法
- iOS - Could not launch"appName" process launch failed: timed out waiting for app to launch:
- Oracle RAC root.sh 报错 Timed out waiting for the CRS stack to start 解决方法
- ios 遇到的错误 ---ios process lunch filed timed out waiting for app to lunch
- iOS - Could not launch"appName" process launch failed: timed out waiting for app to launch
- process launch failed: timed out waiting for app to launch问题解决
- Could not launch "xxx". process launch failed: timed out waiting for app to launch解决方案
- Could not launch "xxx". process launch failed: timed out waiting for app to launch
- iOS真机调试时“process launch failed: timed out waiting for app to launch”问题
- iOS开发 报错process launch failed: timed out waiting for app to launch的解决方案
- 一站式学习Wireshark(五):TCP窗口与拥塞处理
- 解决Apple World Wide Developer Relations Certificate Authority证书过期
- 【Android 应用开发】Activity 状态保存 OnSaveInstanceState参数解析
- github上的一个html5俄罗斯方块小游戏做了些简化
- 使用maven开发spring项目报ClassNotFoundException异常
- MessagingTimeout: Timed out waiting for a reply to message ID
- 一站式学习Wireshark(六):狙击网络高延时点
- maven冲突解决
- 实时渲染
- Java基础:Day01笔记内容 (概述、JDK的下载及安装)
- java.nio.ByteBuffer用法小结
- Android控件
- iOS 新手引导页面
- 一站式学习Wireshark(七):Statistics统计工具功能详解与应用