python multiprocessing share variable

在某次开发中多进程间需要频繁共享变量， 碰到了各种各样的问题，总结一下。

一、multiprocessing中的Queue共享问题

1.先看一段代码

import multiprocessing, timeq = multiprocessing.Queue()def task(count):    for i in xrange(count):        q.put("%d mississippi" % i)    return "Done"def main():    pool = multiprocessing.Pool()    result = pool.apply_async(task, (1,))    time.sleep(1)    while not q.empty():        print q.get()    print result.get()if __name__ == "__main__":    main()

运行之后，输出结果：

0 mississippi
Done

通过以上代码我们知道进程间通信可调用multiprocessing.Queue.但会发现此时的q是global。试想如果是局部变量情况会是怎么样的呢？看下面代码：

import multiprocessing, timedef task(count):    for i in xrange(count):        q.put("%d mississippi" % i)    return "Done"def main():    q = multiprocessing.Queue()    pool = multiprocessing.Pool()    result = pool.apply_async(task, (1, q))     time.sleep(1)    while not q.empty():        print q.get()    print result.get()if __name__ == "__main__":    main()

运行会发现报错。

RuntimeError: Queue objects should only be shared between processes through inheritance

解决方法1》： 在Pool初始化queue

import multiprocessing, timedef task(count):    for i in xrange(count):        task.q.put("%d mississippi" % i)    return "Done"def task_init(q):    task.q = q def main():    q = multiprocessing.Queue()    pool = multiprocessing.Pool(None, task_init, (q,))    result = pool.apply_async(task, (1,))    time.sleep(1)    while not q.empty():        print q.get()    print result.get()if __name__ == "__main__":    main()

解决方法2》：调用multiprocessing下的manager（）中的queue

import multiprocessing, timedef task(count, q):     for i in xrange(count):        q.put("%d mississippi" % i)    return "Done"def main():    q = multiprocessing.Manager().Queue()    pool = multiprocessing.Pool()    result = pool.apply_async(task, (1,q,))    time.sleep(1)    while not q.empty():        print q.get()    print result.get()if __name__ == "__main__":    main()

小结： 多进程间变量共享均可以调用Manager（）下的类型， A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore,Condition, Event, Queue, Value and Array.

 二、调用Pool中的apply, apply_async区别

官方文档上说的比较简单:

apply(func[, args[, kwds]])
Equivalent of the apply() built-in function. It blocks until the result is ready, so apply_async() is better suited for performing work in parallel. Additionally, func is only executed in one of the workers of the pool.
apply_async(func[, args[, kwds[, callback]]])
A variant of the apply() method which returns a result object.
If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callbackis applied to it (unless the call failed). callback should complete immediately since otherwise the thread which handles the results will get blocked.

我大概所理解的意思就是:

apply, 是一个内建函数，当调用函数时会被堵住直到有返回结果。apply_async（）更适合用于平行工作。对于apply而言，func事实上只会在一个连接中工作。

apply_async是apply函数的一个变种，当callback被指定时，它必须是能接受一个单独的参数并且是可调用的。当结果生成时callback将立马被执行除非是返回结果的函数被堵住了。

但是在使用的过程中还是要多注意apply_async所带来的问题。

我所遇到的是异常捕获的问题。看下面一段代码：

from multiprocessing import Manager, Poolfrom collections import namedtupleimport loggingdef test():    A=namedtuple('A', ['x', 'y'])    return A(1, 2)def f(q):    x = test()    try:        q.put(x)    except:        raisetry:    manager = Manager()    q = manager.Queue()    q.put(1)    p = Pool(processes = 2)    for i in range(3):        p.apply_async(f, args=(q,))    p.close()    p.join()    while q.empty() is False:        print q.get()except:    logging.error('55555555', exc_info=True)

我本来的意图是想证明queue共享的问题。期望看到的结果是输出1， A（1，2）.结果输出就只有1.并且没有任何异常抛出。

我一步一步跟进去发现函数f中是出现了异常的，由于namedtuple的也就是A的定义不是全局可见的，所以会抛出异常，但在父进程却没有捕获到。

于是我写了一个很简单的例子来证明父进程是否能捕获到子进程的异常。如下代码：

import multiprocessingdef go():    print '---'    raise Exception()if __name__ == '__main__':    p = multiprocessing.Pool(processes = 1)    p.apply_async(go)    p.close()    p.join()

程序正常输出---之后就结束了，未能捕获到异常。但是将apply_aync换成apply之后就能正常捕获到异常了。想了下apply是会block子进程的，捕获到异常很正常。但是apply_async的异常为什么会报不出来呢？

查看官方文档，看到下面一段话：

class multiprocessing.pool.AsyncResult
The class of the result returned by Pool.apply_async() and Pool.map_async().
get([timeout])
Return the result when it arrives. If timeout is not None and the result does not arrive within timeout seconds then multiprocessing.TimeoutError is raised. If the remote call raised an exception then that exception will be reraised by get().

大概意思就是apply_async的返回值是asyncresult类型，当子进程出现异常时，可以通过get（）函数来捕获。

在网上搜到一段别人的代码能详细捕获异常的代码。

import tracebackfrom multiprocessing.pool import Poolimport multiprocessing# Shortcut to multiprocessing's loggerdef error(msg, *args):    return multiprocessing.get_logger().error(msg, *args)class LogExceptions(object):    def __init__(self, callable):        self.__callable = callable        return    def __call__(self, *args, **kwargs):        try:            result = self.__callable(*args, **kwargs)        except Exception as e:            # Here we add some debugging help. If multiprocessing's            # debugging is on, it will arrange to log the traceback            error(traceback.format_exc())            # Re-raise the original exception so the Pool worker can            # clean up            raise        # It was fine, give a normal answer        return result    passclass LoggingPool(Pool):    def apply_async(self, func, args=(), kwds={}, callback=None):        return Pool.apply_async(self, LogExceptions(func), args, kwds, callback)def go():    print(1)    raise Exception()    print(2)multiprocessing.log_to_stderr()p = LoggingPool(processes=1)p.apply_async(go)p.close()p.join()

三、 PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed 错误

在调用multiprocessing的过程中经常性碰到这个错误，比如以下代码：

import multiprocessing as mpclass Foo():    @staticmethod    def work(self):        passpool = mp.Pool()foo = Foo()pool.apply_async(foo.work)pool.close()pool.join()

查找资料发现一个比较好的解释：

The problem is that the pool methods all use a queue.Queue to pass tasks to the worker processes. Everything that goes through the queue.Queue must be pickable, and foo.work is not picklable since it is not defined at the top level of the module.

解决方法：

def work(foo):    foo.work()pool.apply_async(work,args=(foo,))