故障案例,mongo副本集主节频繁切换
来源:互联网 发布:网络彩票停售原因 编辑:程序博客网 时间:2024/05/06 09:30
接到个case,mongod版本为2.4,三个节点组成高可用,发现经常频繁地切换,错误信息如下
Mon Mar 7 14:29:15.379 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
Mon Mar 7 14:29:15.379 [initandlisten] can't create new thread, closing connection
Mon Mar 7 14:29:15.381 [initandlisten] connection accepted from 10.10.14.71:41282 #1801013 (973 connections now open)
Mon Mar 7 14:29:15.381 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
Mon Mar 7 14:29:15.381 [initandlisten] can't create new thread, closing connection
Mon Mar 7 14:29:15.383 [initandlisten] connection accepted from 10.10.14.71:41270 #1801014 (973 connections now open)
Mon Mar 7 14:29:15.383 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
Mon Mar 7 14:29:15.383 [initandlisten] can't create new thread, closing connection
Mon Mar 7 14:29:15.385 [initandlisten] connection accepted from 10.10.14.71:41283 #1801015 (973 connections now open)
Mon Mar 7 14:29:15.385 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
Mon Mar 7 14:29:15.385 [initandlisten] can't create new thread, closing connection
Mon Mar 7 14:29:19.689 [rsHealthPoll] replSet health poll task caught an exception: boost::thread_resource_errorreplSet info 10.10.223.207:27017 is down (or slow to respond): boost::thread_resource_error
Mon Mar 7 14:29:19.689 [rsHealthPoll] replSet member 10.10.223.207:27017 is now in state DOWN
Mon Mar 7 14:29:21.949 [rsHealthPoll] replSet health poll task caught an exception: boost::thread_resource_errorreplSet info 10.10.14.71:27017 is down (or slow to respond): boost::thread_resource_error
Mon Mar 7 14:29:21.949 [rsHealthPoll] replSet member 10.10.14.71:27017 is now in state DOWN
mongod进程一直都在,后来监控内网质量也无影响。
仔细看错误日志,发现连接数一直上不到1000,而配置的连接数有2w,最后定位到是centos的设置问题
13># cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
* soft nproc 1024
root soft nproc unlimited
发现非root账户限制在了1024,而正好mongod进程是非root用户拉起的,增大该值后,问题得到解决。
- 故障案例,mongo副本集主节频繁切换
- 日志切换频繁解决案例
- 故障案例--mongodb添加副本集成员报错
- 故障案例:mongodb 连接副本集很慢
- 故障案例--mongo备份文件损坏,导致mongorestore中断
- 故障案例--mongo 3.0鉴权导致cpu居高不下
- mongo安装副本集
- 故障案例--mongodb副本集write concern为majority的一个坑
- 故障案例--mongodb副本集成员节点和投票节点的限制
- mongo副本集群搭建过程
- 用Mongos代替驱动来增加副本集的故障切换能力
- 用Mongos代替驱动来增加副本集的故障切换能力
- 故障案例--mongo shell从库无法读的处理方法
- MongDB副本集(一),SpringBoot链接Mongo副本集
- 将mongo副本集转换为分片
- Mongo副本结点的相关配置
- mongo配置副本集(replicaSet)
- mongo shell 高级之 副本集
- HDU 1879 继续畅通工程(Kruskal求最小生成树)
- 基于Jenkins+git+gradle+pgyer的android持续集成
- 51nod 1081 子段求和
- 在iOS开发中如何优雅地进行图片缩放?
- 使用Get和Post提交数据的实现和差别
- 故障案例,mongo副本集主节频繁切换
- Dubbo分布式远程服务调用框架(告别Web Service模式中的WSdl,以服务者与消费者的方式在dubbo上注册
- windows下PostgreSQL打开psql.exe提示口令无法进入
- EJB3方法调用的错误及其处理
- 典型分治问题 棋盘覆盖
- Log4j使用总结(全面)一
- 常用的搜索排序算法总结
- boost::timer::cpu_timer 使用
- Eclipse给Jar文件指定源代码文件