Nova报错：OperationalError: (_mysql_exceptions.OperationalError) (1040, 'Too many connections')

Openstack版本：Liberty

系统版本：CentOS 64bit

##########################################################

1.现象

当在Dashboard上操作一段时间后，会频繁出现报错，使用命令行操作也是出现“500 未知内部错误“，经查nova-api.log发现如下报错：

2016-04-11 14:23:22.069 99072 INFO nova.osapi_compute.wsgi.server [req-96fbde09-2796-4153-8209-ae3a22f4d35d ca23866c0b774f5ca978e50beaa61a96 80fd780584f3427797db539a06c9f60c - - -] 192.168.91.7 "POST /v2/80fd780584f3427797db539a06c9f60c/servers/5cda35ed-4211-44eb-ad82-c29a9be5c657/action HTTP/1.1" status: 200 len: 317 time: 0.10823512016-04-11 14:23:22.178 99072 INFO nova.osapi_compute.wsgi.server [req-0a30a628-a436-4779-9cb4-71d6aa947a50 ca23866c0b774f5ca978e50beaa61a96 80fd780584f3427797db539a06c9f60c - - -] 192.168.91.7 "POST /v2/80fd780584f3427797db539a06c9f60c/servers/d04bd833-db6d-4adf-98d7-3c37f612009b/action HTTP/1.1" status: 200 len: 317 time: 0.10459092016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions [req-b69f6cac-6905-41c9-8a0c-abf733ba4178 ca23866c0b774f5ca978e50beaa61a96 80fd780584f3427797db539a06c9f60c - - -] Unexpected exception in API method2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions Traceback (most recent call last):2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/remote_consoles.py", line 53, in get_vnc_console2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     instance = common.get_instance(self.compute_api, context, id)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/common.py", line 510, in get_instance2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     expected_attrs=expected_attrs)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2026, in get2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     context, instance_id, expected_attrs=expected_attrs)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 171, in wrapper2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 372, in get_by_uuid2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     use_slave=use_slave)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 645, in instance_get_by_uuid2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     columns_to_join, use_slave=use_slave)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 216, in wrapper2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1713, in instance_get_by_uuid2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join, use_slave=use_slave)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1720, in _instance_get_by_uuid2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     use_slave=use_slave).\2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1751, in _build_instance_get2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     project_only=True, use_slave=use_slave).\2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 275, in model_query2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     session = get_session(use_slave=use_slave)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 182, in get_session2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     facade = _create_facade_lazily(_MAIN_FACADE, conf_group)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 164, in _create_facade_lazily2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     _ENGINE_FACADE[facade] = _create_facade(conf_group)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 156, in _create_facade2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     retry_interval=conf_group.retry_interval)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 943, in __init__2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     slave_connection=slave_connection)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     engine_args, maker_args)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     sql_connection=sql_connection, **engine_kwargs)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     test_conn = _test_connection(engine, max_retries, retry_interval)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 326, in _test_connection2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return engine.connect()2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2018, in connect2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return self._connection_cls(self, **kwargs)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in __init__2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     if connection is not None else engine.raw_connection()2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     self.pool.unique_connection, _connection)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     e, dialect, self)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     util.raise_from_cause(newraise, exc_info)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return fn()2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return _ConnectionFairy._checkout(self)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     fairy = _ConnectionRecord.checkout(pool)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     rec = pool._do_get()2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     self._dec_overflow()2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     compat.reraise(exc_type, exc_value, exc_tb)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return self._create_connection()2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return _ConnectionRecord(self)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     self.connection = self.__connect()2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     connection = self.__pool._invoke_creator(self)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return dialect.connect(*cargs, **cparams)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return self.dbapi.connect(*cargs, **cparams)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     return Connection(*args, **kwargs)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions     super(Connection, self).__init__(*args, **kwargs2)2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions OperationalError: (_mysql_exceptions.OperationalError) (1040, 'Too many connections')2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions 2016-04-11 14:23:22.639 99106 INFO nova.api.openstack.wsgi [req-b69f6cac-6905-41c9-8a0c-abf733ba4178 ca23866c0b774f5ca978e50beaa61a96 80fd780584f3427797db539a06c9f60c - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.<class 'sqlalchemy.exc.OperationalError'>

####################################################################

2.问题解决

根据报错提示，显然是数据库超最大连接了，调整mariadb数据库最大连接即可(nova连接数据库采用的是永久连接，永久连接数大约设在200~300之间，这块源码我还没有细看，大家可以根据需要自行调整数据库最大连接数大于300即可)。

3.注意

1.CentOS7系统调整数据库最大连接数和之前有所不同，具体可参考：

CentOS7.2 调整mysql数据库最大连接数