celery+Rabbit MQ的安装和使用
来源:互联网 发布:有个卖时间的软件 编辑:程序博客网 时间:2024/05/29 04:37
celery 和Rabbit MQ作为异步任务处理的组合,在生产中得到广泛的使用。本文简单介绍其安装和使用方法。
安装 RabbitMQ
安装erlang
RabbitMQ 依赖erlang环境,首先安装erlang环境,如果没有安装会报错
erl: command not found
下载安装包otp_src_19.0.tar.gz
https://www.erlang.org/downloads/19.0
解压,配置,编译,安装
$ ./configure$ make$ make install
安装RabbitMQ
对于centos 6,
https://www.rabbitmq.com/install-generic-unix.html
解压即可使用
$ cd rabbitmq_server-3.6.10# 启动服务$ sbin/rabbitmq-server
启动为后台服务(只有一个破折号)
$ sbin/rabbitmq-server -detached
查看状态
$ sbin/rabbitmqctl status
停止服务
永远不要用 kill 停止 RabbitMQ 服务器,而是应该用 rabbitmqctl 命令:
$ sbin/rabbitmqctl stop
默认用户guest,密码guest,只允许本地访问,如需远程访问,需要设置;
添加用户
$ sbin/rabbitmqctl add_user test 123456Creating user "test"
添加虚拟主机,并赋予用户test权限
$ sbin/rabbitmqctl add_vhost myvhostCreating vhost "myvhost"$ sbin/rabbitmqctl set_permissions -p myvhost test ".*" ".*" ".*"Setting permissions for user "test" in vhost "myvhost"
安装celery
使用pip安装celery
$ pip install celery
使用celery
创建celery实例,指定task及MQ地址。
# tasks.pyfrom celery import Celeryapp = Celery('tasks', broker='amqp://test:123456@localhost/myvhost')@app.taskdef add(x, y): return x + y
启动celery
在tasks.py文件所在目录,执行如下命令:
$ /data/happy_env/bin/celery worker -A tasks --loglevel=info -------------- celery@localhost.localdomain v4.1.0 (latentcall)---- **** ----- --- * *** * -- Linux-2.6.32-642.el6.x86_64-x86_64-with-centos-6.8-Final 2017-08-05 17:20:25-- * - **** --- - ** ---------- [config]- ** ---------- .> app: tasks:0x7f1e73342190- ** ---------- .> transport: amqp://test:**@localhost:5672/myvhost- ** ---------- .> results: disabled://- *** --- * --- .> concurrency: 4 (prefork)-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)--- ***** ----- -------------- [queues] .> celery exchange=celery(direct) key=celery[tasks] . tasks.add[2017-08-05 17:20:25,519: INFO/MainProcess] Connected to amqp://test:**@127.0.0.1:5672/myvhost[2017-08-05 17:20:25,535: INFO/MainProcess] mingle: searching for neighbors[2017-08-05 17:20:26,576: INFO/MainProcess] mingle: all alone[2017-08-05 17:20:26,692: INFO/MainProcess] celery@localhost.localdomain ready.
其中,
- –app or -A 指定app实例名称
- –loglevel or -l指定日志打印级别
默认使用的queue名称是celery.
调用任务
所谓调用任务,就是向MQ放消息,worker从MQ取消息,进行消费。
进入python 终端:
$ /data/happy_env/bin/python Python 2.7.13 (default, Aug 5 2017, 14:56:29) [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> from tasks import add>>> add.delay(6, 6)<AsyncResult: 6f270cbd-4e90-4d21-842c-50f8488f8216>>>>
add.delay(6,6)相当于add.apply_async((2, 2)),默认的queue是celery。
apply_async还可以指定queue等参数:
>>> add.apply_async((2, 2), queue='lopri', countdown=10)
celery worker终端输出:
[2017-08-05 17:32:37,289: INFO/MainProcess] Received task: tasks.add[6f270cbd-4e90-4d21-842c-50f8488f8216]
[2017-08-05 17:32:37,300: INFO/ForkPoolWorker-3] Task tasks.add[6f270cbd-4e90-4d21-842c-50f8488f8216] succeeded in 0.0021863639995s: 12
从文件中读取配置
可以将配置写到文件中。
#tasks.pyfrom celery import Celeryapp = Celery("orange")app.config_from_object("celeryconfig")@app.taskdef add(x, y): return x + y
配置文件celeryconfig.py内容:
RABBIT_MQ = { 'HOST': '127.0.0.1', 'PORT': 5672, 'USER': 'test', 'PASSWORD': '123456'}# brokerBROKER_URL = 'amqp://%s:%s@%s:%s/myvhost' % (RABBIT_MQ['USER'], RABBIT_MQ['PASSWORD'], RABBIT_MQ['HOST'], RABBIT_MQ['PORT'])# celery日志格式CELERYD_LOG_FORMAT = '[%(asctime)s] [%(levelname)s] %(message)s'
配置celery日志保存文件
$ /data/happy_env/bin/celery worker -A tasks --loglevel=info -f /home/lanyang/python_ex/celery_test/test.log
保存结果
后端通过 Celery 的 backend 参数来指定。如果你选择使用配置模块,则通过 CELERY_RESULT_BACKEND 选项来设置。
routing
默认情况下,worker从queue名称为celery的队列中消费消息。可以通过定义CELERY_ROUTES配置项来指定queue。
配置文件celeryconfig.py:
RABBIT_MQ = { 'HOST': '127.0.0.1', 'PORT': 5672, 'USER': 'test', 'PASSWORD': '123456'}BROKER_URL = 'amqp://%s:%s@%s:%s/myvhost' % (RABBIT_MQ['USER'], RABBIT_MQ['PASSWORD'], RABBIT_MQ['HOST'], RABBIT_MQ['PORT'])CELERYD_LOG_FORMAT = '[%(asctime)s] [%(levelname)s] %(message)s'CELERY_ROUTES = { 'tasks.add': {'queue': 'sunday'},}
启动生产者
>>> from tasks import add>>> add.apply_async((2, 2))
配置文件中的queue是sunday,所以是向sunday发送消息;
启动消费者celery worker
$ /data/happy_env/bin/celery worker -A tasks --loglevel=info –Q sunday -f /home/lanyang/python_ex/celery_test/test.log-------------- celery@localhost.localdomain v4.1.0 (latentcall)---- **** ----- --- * *** * -- Linux-2.6.32-642.el6.x86_64-x86_64-with-centos-6.8-Final 2017-08-06 11:14:35-- * - **** --- - ** ---------- [config]- ** ---------- .> app: assetv2:0x7f76f3049590- ** ---------- .> transport: amqp://test:**@127.0.0.1:5672/myvhost- ** ---------- .> results: disabled://- *** --- * --- .> concurrency: 4 (prefork)-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)--- ***** ----- -------------- [queues] .> sunday exchange=sunday(direct) key=sunday[tasks] . tasks.add
其中,
- -Q指定从这个queue中消费消息
- -f 指定log文件
log文件:
[2017-08-06 11:14:35,940] [INFO] Connected to amqp://test:**@127.0.0.1:5672/myvhost
[2017-08-06 11:14:35,952] [INFO] mingle: searching for neighbors
[2017-08-06 11:14:37,019] [INFO] mingle: all alone
[2017-08-06 11:14:37,033] [INFO] celery@localhost.localdomain ready.
[2017-08-06 11:14:37,034] [INFO] Received task: tasks.add[384b604f-ed50-4066-be3a-681120520d1a]
[2017-08-06 11:14:37,136] [INFO] logger this is a message from arm…..
[2017-08-06 11:14:37,137] [INFO] logging from arm…..
[2017-08-06 11:14:37,137] [WARNING] logger warning….
[2017-08-06 11:14:37,137] [INFO] Task tasks.add[384b604f-ed50-4066-be3a-681120520d1a] succeeded in 0.00111318299969s: 4
Exchanges, queues and routing keys
官网文档如下:
1. Messages are sent to exchanges.
2. An exchange routes messages to one or more queues. Several exchange types exists, providing different ways to do routing, or implementing different messaging scenarios.根据route_key,找到对应queue。celery中的queue就是此处的route_key.
3. The message waits in the queue until someone consumes it.
4. The message is deleted from the queue when it has been acknowledged.
The steps required to send and receive messages are:
- .Create an exchange
- Create a queue
- Bind the queue to the exchange.
Celery automatically creates the entities necessary for the queues in CELERY_QUEUES to work (默认True,except if the queue’s auto_declare setting is set to False).
关于Exchanges, queues and routing keys,翻译一下,大体关系如下:
- 消息被发送到exchage
- 关联到具体的queue
- 消息被消费之前一直存在于queue中
- 消息被消费后,自动从queue中删除
生产和消费消息的过程如下:
- 创建exchange
- 创建queue
- 将queue绑定到exchange
celery会自动创建exchange、queue,并把queue绑定到exchange。
更多信息
http://docs.jinkan.org/docs/celery/userguide/routing.html
参考
celery
http://docs.jinkan.org/docs/celery/getting-started/first-steps-with-celery.html#first-steps
rabbitMQ
https://www.rabbitmq.com/install-generic-unix.html
关于celery 精灵化
http://docs.jinkan.org/docs/celery/tutorials/daemonizing.html#centos
supervisor+celery+celerybeat入门指南
https://github.com/importcjj/notes/issues/2
- celery+Rabbit MQ的安装和使用
- Rabbit MQ 安装和配置
- Celery的安装和使用
- Celery、flower和celery-redis 的安装与使用
- 安装Rabbit MQ
- window 安装Rabbit MQ
- Rabbit Mq 异步消息的使用
- celery-redis的安装和使用
- linux环境 Rabbit mq安装
- Windows 上安装Rabbit MQ
- Rabbit MQ windows下安装
- windows下的php rabbit mq安装、配置
- 在windows上安装Rabbit MQ的详细步骤
- Flask和Celery的使用
- celery的配置和使用
- rabbit mq
- rabbit MQ
- rabbit mq
- 25个强大的CSS代码
- 约瑟环解法
- spring IOC和DI
- 31. Next Permutation
- HDOJ1176 免费馅饼 简单的二维DP
- celery+Rabbit MQ的安装和使用
- 20170811
- 洛谷3379(LCA模板优化)
- 单行注释、多行注释、文档注释最完美的解释、导出API
- Java_23种设计模式(一)
- 笔记本无缘无故自动唤醒是怎么回事?
- 虚拟机CentOS7安装python3.6.2及requests模块的问题汇总
- Android中MediaCodec的使用
- 51nod-1434 区间LCM