mongodb 分片集群thread_resource_error故障

来源：互联网发布：邮箱数据吧编辑：程序博客网时间：2024/06/06 00:54

1、故障记录如下：

Wed Sep 28 14:07:46.446 [Balancer] caught exception while doing balance: socket exception [CONNECT_ERROR] for 10.254.3.63:20000
Wed Sep 28 14:07:47.716 [conn1165] DBClientCursor::init call() failed
Wed Sep 28 14:07:47.716 [conn1165] warning: problem while initially checking shard versions on shard1 :: caused by :: DBClientBase::findN: transport error: 10.254.3.62:27017 ns: admin.$cmd query: { setShardVersion: "", init: true, configdb: "10.254.3.62:20000,10.254.3.63:20000,10.254.3.72:20000", serverID: ObjectId('57e0c83c3d42876f0f641669'), authoritative: true }
Wed Sep 28 14:07:47.731 [conn1165] warning: db exception when initializing on shard2:shard2/10.254.3.62:27018,10.254.3.72:27028, current connection state is { state: { conn: "", vinfo: "shard2:shard2/10.254.3.62:27018,10.254.3.72:27028", cursor: "(none)", count: 0, done: false }, retryNext: false, init: false, finish: false, errored: false } :: caused by :: 10276 DBClientBase::findN: transport error: 10.254.3.62:27017 ns: admin.$cmd query: { setShardVersion: "", init: true, configdb: "10.254.3.62:20000,10.254.3.63:20000,10.254.3.72:20000", serverID: ObjectId('57e0c83c3d42876f0f641669'), authoritative: true }
Wed Sep 28 14:07:52.447 [Balancer] trying reconnect to 10.254.3.63:20000
Wed Sep 28 14:07:55.450 [Balancer] reconnect 10.254.3.63:20000 failed couldn't connect to server 10.254.3.63:20000
Wed Sep 28 14:07:55.450 [CheckConfigServers] warning: couldn't check on config server:10.254.3.63:20000 ok for now : 11002 socket exception [6] server [10.254.3.63:20000] mongos connectionpool error: couldn't connect to server 10.254.3.63:20000
Wed Sep 28 14:07:55.450 [Balancer] SyncClusterConnection connecting to [10.254.3.62:20000]
Wed Sep 28 14:07:55.451 [Balancer] SyncClusterConnection connecting to [10.254.3.63:20000]
Wed Sep 28 14:07:58.468 [Balancer] SyncClusterConnection connect fail to: 10.254.3.63:20000 errmsg: couldn't connect to server 10.254.3.63:20000
Wed Sep 28 14:07:58.468 [Balancer] SyncClusterConnection connecting to [10.254.3.72:20000]
Wed Sep 28 14:07:59.408 [conn1165] warning: exception when initializing on shard2:shard2/10.254.3.62:27018,10.254.3.72:27028, current connection state is { state: { conn: "", vinfo: "shard2:shard2/10.254.3.62:27018,10.254.3.72:27028", cursor: "(none)", count: 0, done: false }, retryNext: false, init: false, finish: false, errored: false } :: caused by :: boost::thread_resource_error
Wed Sep 28 14:07:59.408 [conn1165] ERROR: Uncaught std::exception: boost::thread_resource_error, terminating
Wed Sep 28 14:07:59.408 [conn1165] dbexit: rc:100

***** SERVER RESTARTED *****
Wed Sep 28 14:57:39.361 [mongosMain] MongoS version 2.4.4 starting: pid=60710 port=30000 64-bit host=db_m1_slave_1 (--help for usage)
Wed Sep 28 14:57:39.361 [mongosMain] git version: 4ec1fb96702c9d4c57b1e06dd34eb73a16e407d2
Wed Sep 28 14:57:39.361 [mongosMain] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Sep 28 14:57:39.361 [mongosMain] options: { chunkSize: 64, configdb: "10.254.3.62:20000,10.254.3.63:20000,10.254.3.72:20000", fork: true, logappend: true, logpath: "/data/mongodb/logs_244/mongos5.log", port: 30000 }
Wed Sep 28 14:57:42.378 [mongosMain] warning: couldn't check on config server:10.254.3.63:20000 ok for now : 11002 socket exception [6] server [10.254.3.63:20000] mongos connectionpool error: couldn't connect to server 10.254.3.63:20000

2、解决方案

重启mongos服务

3 0