{转}dispy,asyncoro实现的分布式并行计算框架
来源:互联网 发布:三国志9优化伴侣32位 编辑:程序博客网 时间:2024/06/14 14:21
{转}dispy,asyncoro实现的分布式并行计算框架
dispy:asyncoro实现的分布式并行计算框架。一个对asyncoro很有说明性的案例。
框架也是非常精简,只有4个组件
dispy.py (client) provides two ways of creating "clusters": JobCluster when only one instance of dispy may run and SharedJobCluster when multiple instances may run (in separate processes). If JobCluster is used, the scheduler contained within dispy.py will distribute jobs on the server nodes; if SharedJobCluster is used, a separate scheduler (dispyscheduler) must be running.
dispynode.py executes jobs on behalf of dispy. dispynode must be running on each of the (server) nodes that form the cluster.
dispyscheduler.py is needed only when SharedJobCluster is used; this provides a scheduler that can be shared by multiple dispy users.
dispynetrelay.py is needed when nodes are located across different networks; this relays information about nodes on a network to the scheduler. If all the nodes are on same network, there is no need for dispynetrelay - the scheduler and nodes automatically discover each other.
一般情况下,使用dispy和dispynode就已经足够解决问题了
- 服务端:
dispynode是服务端组件,它不需要写代码,只是使用参数运行为一个守护进程就OK了,比如:
dispynode.py -c 2 -i 192.168.0.10 -p 51348 -s secret
这个实例会使用2个cpu核心,绑定192.168.0.10:51348地址提供服务,secret是消息加密的共享密钥,更多参数参见http://dispy.sourceforge.net/dispynode.html
- 客户端:
简单例子:
#!/usr/bin/env pythondef compute(n): import time, socket time.sleep(n) host = socket.gethostname() return (host, n)if __name__ == '__main__': import dispy, random cluster = dispy.JobCluster(compute,nodes=['192.168.0.10', '192.168.3.11']) jobs = [] for n in range(20): job = cluster.submit(random.randint(5,20)) job.id = n jobs.append(job) # cluster.wait() for job in jobs: host, n = job() print '%s executed job %s at %s with %s' % (host, job.id, job.start_time, n) # other fields of 'job' that may be useful: # print job.stdout, job.stderr, job.exception, job.ip_addr, job.start_time, job.end_time cluster.stats()
JobCluster也可以使用回调来处理结果:
dispy.JobCluster(compute,nodes=['192.168.0.10', '192.168.3.11'],callback=callback)
除了python函数,也可以是调用服务端的程序,比如:
cluster = dispy.JobCluster('/some/program', nodes=['192.168.0.10'])
也可以不写任何代码,而作为一个命令工具来使用:
dispy.py -f /some/file1 -f file2 -a "arg11 arg12" -a "arg21 arg22" -a "arg3" /some/program
详细文档参见 http://dispy.sourceforge.net/index.html
如果嫌celery稍重的话,可以试试dispy
- {转}dispy,asyncoro实现的分布式并行计算框架
- dispy,asyncoro实现的分布式并行计算框架
- dispy,asyncoro实现的分布式并行计算框架
- dispy,asyncoro实现的分布式并行计算框架
- python分布式计算dispy简单使用
- Gearman 并行分布式计算框架
- 打造多合一的分布式并行计算框架
- 分布式文件系统,NoSQL数据库,并行计算框架
- Fourinone分布式并行计算四合一框架
- lucene+hadoop 分布式并行计算搜索框架
- 分布式图并行计算框架:PowerGraph
- 分布式计算和并行计算的异同
- 分布式计算和并行计算的异同
- 并行计算与分布式计算的区别
- 分布式计算和并行计算的区别
- 巧用zookeeper实现分布式并行计算
- 巧用zookeeper实现分布式并行计算
- 并行计算,分布式计算
- qt-embedded-4.5.2移植实验
- cocos2dx3.3 便条提示语功能简单实现
- 高性能网站建设-14条规则
- [Leetcode] 32 - Longest Valid Parentheses
- android TextView多行文本(超过3行)使用ellipsize属性无效
- {转}dispy,asyncoro实现的分布式并行计算框架
- Windows Server 2008 R2不能使用指纹识别、蓝牙
- hdu 4561
- codeforces 189A Cut Ribbon
- AVL树
- 几种基本工作流模式
- Android控件源码分析--AndroidResideMenu菜单
- 新东西001--android数据库升级维护
- 【Dokan学习记录】2015-01-29