Parallel Python 并行计算

原文链接 http://www.cnblogs.com/flyingis/archive/2009/11/12/1601574.html
作者： Flyingis

最近在关注如何提升Python执行效率的问题，自己没有时间去深入研究，就直接选择了开源的Parallel Python，希望能够充分发挥多核CPU及集群环境的优势。
Parallel Python是Python进行分布式计算的开源模块，能够将计算压力分布到多核CPU或集群的多台计算机上，能够非常方便的在内网中搭建一个自组织的分布式计算平台。先从多核计算开始，普通的Python应用程序只能够使用一个CPU进程，而通过Parallel Python能够很方便的将计算扩展到多个CPU进程中，使用官方网站上的一个例子。

不使用Parallel Python

import math, timedef isprime(n):    """Returns True if n is prime and False otherwise"""    if not isinstance(n, int):        raise TypeError("argument passed to is_prime is not of 'int' type")    if n < 2:        return False    if n == 2:        return True    max = int(math.ceil(math.sqrt(n)))    i = 2    while i <= max:        if n % i == 0:            return False        i += 1    return Truedef sum_primes(n):    """Calculates sum of all primes below given integer n"""    return sum([x for x in xrange(2,n) if isprime(x)])start_time = time.time()inputs = (100000, 100100, 100200, 100300, 100400, 100500, 100600, 100700)jobs = [(input, sum_primes(input)) for input in inputs]for input, job in jobs:    print "Sum of primes below", input, "is", jobprint "Time elapsed: ", time.time() - start_time, "s"

运行结果

使用Parallel Python

将程序稍作调整，引入pp模块。

#!# File: sum_primes.py# Author: VItalii Vanovschi# Desc: This program demonstrates parallel computations with pp module# It calculates the sum of prime numbers below a given integer in parallel# Parallel Python Software: http://www.parallelpython.com/import math, sys, timeimport ppdef isprime(n):    """Returns True if n is prime and False otherwise"""    if not isinstance(n, int):        raise TypeError("argument passed to is_prime is not of 'int' type")    if n < 2:        return False    if n == 2:        return True    max = int(math.ceil(math.sqrt(n)))    i = 2    while i <= max:        if n % i == 0:            return False        i += 1    return Truedef sum_primes(n):    """Calculates sum of all primes below given integer n"""    return sum([x for x in xrange(2,n) if isprime(x)])print """Usage: python sum_primes.py [ncpus]    [ncpus] - the number of workers to run in parallel,     if omitted it will be set to the number of processors in the system"""# tuple of all parallel python servers to connect withppservers = ()#ppservers = ("10.0.0.1",)if len(sys.argv) > 1:    ncpus = int(sys.argv[1])    # Creates jobserver with ncpus workers    job_server = pp.Server(ncpus, ppservers=ppservers)else:    # Creates jobserver with automatically detected number of workers    job_server = pp.Server(ppservers=ppservers)print "Starting pp with", job_server.get_ncpus(), "workers"# Submit a job of calulating sum_primes(100) for execution. # sum_primes - the function# (100,) - tuple with arguments for sum_primes# (isprime,) - tuple with functions on which function sum_primes depends# ("math",) - tuple with module names which must be imported before sum_primes execution# Execution starts as soon as one of the workers will become availablejob1 = job_server.submit(sum_primes, (100,), (isprime,), ("math",))# Retrieves the result calculated by job1# The value of job1() is the same as sum_primes(100)# If the job has not been finished yet, execution will wait here until result is availableresult = job1()print "Sum of primes below 100 is", resultstart_time = time.time()# The following submits 8 jobs and then retrieves the resultsinputs = (100000, 100100, 100200, 100300, 100400, 100500, 100600, 100700)jobs = [(input, job_server.submit(sum_primes,(input,), (isprime,), ("math",))) for input in inputs]for input, job in jobs:    print "Sum of primes below", input, "is", job()print "Time elapsed: ", time.time() - start_time, "s"job_server.print_stats()

运行结果