同一节点启动多个bigchaindb实例

需求

某个节点处于两个不同的bigchaindb集群，或者说一个节点需要启动两个不同的bigchaindb实例

问题

当前bigchaindb版本

BigchainDB (1.0.0rc1)bigchaindb-driver (0.3.1)

原本的bigchiandb配置文件（/root/.bigchaindb）

{    "database": {        "connection_timeout": 5000,        "max_tries": 3,        "replicaset": "bigchain-rs",        "name": "bigchain",        "host": "localhost",        "login": null,        "password": null,        "ssl": false,        "port": 27017,        "backend": "mongodb"    },    "log": {        "level_logfile": "info",        "datefmt_logfile": "%Y-%m-%d %H:%M:%S",        "granular_levels": {},        "file": "/root/bigchaindb.log",        "fmt_logfile": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",        "error_file": "/root/bigchaindb-errors.log",        "fmt_console": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",        "level_console": "info",        "datefmt_console": "%Y-%m-%d %H:%M:%S"    },    "keypair": {        "private": "BAjL63tqdknLbhXB9vP9F9sgaA699z1CgsMKSKjzFsAg",        "public": "PKEsj5XCPZjbbsRraixPriPdJ3j9wm1GrMxkcNUAR5n"    },    "wsserver": {        "port": 9985,        "host": "localhost"    },    "server": {        "threads": null,        "workers": null,        "loglevel": "info",        "bind": "localhost:9984"    },    "keyring": [],    "backlog_reassign_delay": 120}

利用该配置文件启动一个实例

bigchaindb -c /root/.bigchaindb start

复制一份配置文件，修改其中需要的端口（9984->9982,9985->9983），

cp .bigchaindb .bigchaindb2sed -i "s/9985/9983/g" .bigchaindb2sed -i "s/9984/9982/g" .bigchaindb2

再启动一个实例时报错：

root@clean:~# bigchaindb -c .bigchaindb2 startINFO:bigchaindb.config_utils:Configuration loaded from `.bigchaindb2`Traceback (most recent call last):  File "/usr/local/bin/bigchaindb", line 11, in <module>    sys.exit(main())  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/commands/bigchaindb.py", line 348, in main    utils.start(create_parser(), sys.argv[1:], globals())  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/commands/utils.py", line 204, in start    return func(args)  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/commands/utils.py", line 49,in configure    command(args)  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/commands/utils.py", line 75,in start_logging    setup_logging(user_log_config=config.get('log'))  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/log/setup.py", line 49, in setup_logging    setup_sub_logger(user_log_config=user_log_config)  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/log/setup.py", line 38, in setup_sub_logger    server = LogRecordSocketServer()  File "/usr/local/lib/python3.4/dist-packages/bigchaindb/log/setup.py", line 156, in __init__    super().__init__((host, port), handler)  File "/usr/lib/python3.4/socketserver.py", line 430, in __init__    self.server_bind()  File "/usr/lib/python3.4/socketserver.py", line 444, in server_bind    self.socket.bind(self.server_address)OSError: [Errno 98] Address already in useroot@clean:~#

方案

定位到错误位置：/usr/local/lib/python3.4/dist-packages/bigchaindb/log/setup.py

log的setup函数为setup_logging，其中启动了publisher与subscriber。

def setup_logging(*, user_log_config=None):    setup_pub_logger()    setup_sub_logger(user_log_config=user_log_config)

log的publisher调用logging.handlers.SocketHandler将远程输出日志到TCP/IP sockets。注意其中使用了写死的端口DEFAULT_SOCKET_LOGGING_PORT

DEFAULT_SOCKET_LOGGING_HOST = 'localhost'DEFAULT_SOCKET_LOGGING_PORT = DEFAULT_TCP_LOGGING_PORTdef setup_pub_logger():    dictConfig(PUBLISHER_LOGGING_CONFIG)    socket_handler = logging.handlers.SocketHandler(        DEFAULT_SOCKET_LOGGING_HOST, DEFAULT_SOCKET_LOGGING_PORT)    socket_handler.setLevel(logging.DEBUG)    logger = logging.getLogger()    logger.addHandler(socket_handler)

DEFAULT_TCP_LOGGING_PORT在logging模块handlers.py中定义

DEFAULT_TCP_LOGGING_PORT    = 9020

同时，对于日志的subscriber来说，bigchindb启动一个进程来启动subscriber，进程的target为LogRecordSocketServer.serve_forever

def setup_sub_logger(*, user_log_config=None):    server = LogRecordSocketServer()    with server:        server_proc = Process(            target=server.serve_forever,            kwargs={'log_config': user_log_config},        )        server_proc.start()

该类的__init__里实际上已经指定了订阅的端口，即logging.handlers.DEFAULT_TCP_LOGGING_PORT

class LogRecordSocketServer(ThreadingTCPServer):    """    Simple TCP socket-based logging server.    """    allow_reuse_address = True    def __init__(self,                 host='localhost',                 port=logging.handlers.DEFAULT_TCP_LOGGING_PORT,                 handler=LogRecordStreamHandler):        super().__init__((host, port), handler)    def serve_forever(self, *, poll_interval=0.5, log_config=None):        sub_logging_config = create_subscriber_logging_config(            user_log_config=log_config)        dictConfig(sub_logging_config)        try:            super().serve_forever(poll_interval=poll_interval)        except KeyboardInterrupt:            pass

至此，我们可以看到，虽然我们bigchaindb的配置文件进行了修改，使得两个实例的bigchaindb端口没有被占用，但实际上两个实例的log的publisher都采用同一个端口

commands/utils.py调用了setup_logging函数

def start_logging_process(command):    @functools.wraps(command)    def start_logging(args):        from bigchaindb import config        setup_logging(user_log_config=config.get('log'))        command(args)    return start_logging

start_logging_process由run_start通过注解的方式进行调用，并同时调用了configure_bigchaindb，继续跟踪，bigchaindb依次调用了bigchaindb.config_utils.autoconfigure->env_config

@configure_bigchaindb@start_logging_processdef run_start(args):

根据env_config的注释，该函数将config配置文件加载为map，判断是否存在对应的环境变量，否则赋值为配置文件中的值，并最后将配置信息存储在bigchaindb.config中。至此，我们则可以通过在配置文件中加入一个用来配置publisher与subscriber的端口，再在代码中调用bigchaindb.config即可

解决

修改.bigchaindb配置文件，在key值为log下再增加一项

{    "database": {       ...    },    "log": {        "port": 9986,        "level_logfile": "info",    ...

修改

def setup_logging(*, user_log_config=None):    logging.info(">>> publisher port is")    try:        import bigchaindb        global DEFAULT_SOCKET_LOGGING_PORT        DEFAULT_SOCKET_LOGGING_PORT = bigchaindb.config['log']['port']    except:        pass    logging.info(DEFAULT_SOCKET_LOGGING_PORT)    setup_pub_logger()    setup_sub_logger(user_log_config=user_log_config)

def setup_sub_logger(*, user_log_config=None):    server = LogRecordSocketServer(port=DEFAULT_SOCKET_LOGGING_PORT)    with server:        server_proc = Process(            target=server.serve_forever,            kwargs={'log_config': user_log_config},        )        server_proc.start()

再复制一份配置文件：

cp .bigchaindb .bigchaindb2sed -i "s/9984/9981/g" .bigchaindb2sed -i "s/9985/9982/g" .bigchaindb2sed -i "s/9986/9983/g" .bigchaindb2# 修改数据库名sed -i "s/\(\"name\": \"\).*/\1bigchain2\"/g" .bigchaindb2

可以启动两个bigchaindb实例了

bigchaindb -c .bigchaindb startbigchaindb -c .bigchaindb2 start

使用python连接时分别用9984与9981端口即可