RabbitMQ工作队列之Python实现

在上一篇中我们介绍了经典的案例Hello,world。本篇将介绍工作队列。

消息也可以理解为任务，消息发送者可以理解为任务分配者，消息接收者可以理解为工作者，当工作者接收到一个任务，还没完成的时候，任务分配者又发一个任务过来，那就忙不过来了，于是就需要多个工作者来共同处理这些任务，这些工作者，就称为工作队列。结构图如下：

                                                                            

准备工作

在之前的例子中我们发送了一条消息“Hello World”,现在我们发送字符串来模拟复杂的任务，我们通过time.sleep()函数实现，小数点的个数代表任务复杂度.

修改send.py 代码，并重新命名new_task.py:

import sysmessage = ' '.join(sys.argv[1:]) or "Hello World!"channel.basic_publish(exchange='',                      routing_key='hello',                      body=message)print " [x] Sent %r" % (message,)

修改receive.py代码，并重新命名work.py:

import timedef callback(ch, method, properties, body):    print " [x] Received %r" % (body,)    time.sleep( body.count('.') )    print " [x] Done"<span class="s"></span>

循环调度

打开一个终端运行new_task.py：

$ python new_task.py First message.$ python new_task.py Second message..$ python new_task.py Third message...$ python new_task.py Fourth message....$ python new_task.py Fifth message.....

打开两个终端运行work.py：

$ python worker.py [*] Waiting for messages. To exit press CTRL+C [x] Received 'First message.' [x] Received 'Third message...' [x] Received 'Fifth message.....'

$ python worker.py [*] Waiting for messages. To exit press CTRL+C [x] Received 'Second message..' [x] Received 'Fourth message....'

消息确认

消息确认就是当工作者完成任务后，会反馈给rabbitmq。修改worker.py中的回调函数：

def callback(ch, method, properties, body):    print " [x] Received %r" % (body,)    time.sleep( body.count('.') )    print " [x] Done"    ch.basic_ack(delivery_tag = method.delivery_tag)channel.basic_consume(callback,queue='hello')

消息持久化

虽然有了消息反馈机制，但是如果rabbitmq自身挂掉的话，那么任务还是会丢失。所以需要将任务持久化存储起来。声明持久化存储：

channel.queue_declare(queue='task_queue', durable=True)

在发送任务的时候，用delivery_mode=2来标记任务为持久化存储：

channel.basic_publish(exchange='',                      routing_key="task_queue",                      body=message,                      properties=pika.BasicProperties(                         delivery_mode = 2, # make message persistent                      ))

公平调度

虽然每个工作者是依次分配到任务，但是每个任务不一定一样。可能有的任务比较重，执行时间比较久；有的任务比较轻，执行时间比较短。如果能公平调度就最好了，使用basic_qos设置prefetch_count=1，使得rabbitmq不会在同一时间给工作者分配多个任务，即只有工作者完成任务之后，才会再次接收到任务。

                                                

channel.basic_qos(prefetch_count=1)

完整代码：

new_task.py

import pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.queue_declare(queue='task_queue', durable=True)message = ' '.join(sys.argv[1:]) or "Hello World!"channel.basic_publish(exchange='',                      routing_key='task_queue',                      body=message,                      properties=pika.BasicProperties(                         delivery_mode = 2, # make message persistent                      ))print " [x] Sent %r" % (message,)connection.close()

work.py

#!/usr/bin/env pythonimport pikaimport timeconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.queue_declare(queue='task_queue', durable=True)print ' [*] Waiting for messages. To exit press CTRL+C'def callback(ch, method, properties, body):    print " [x] Received %r" % (body,)    time.sleep( body.count('.') )    print " [x] Done"    ch.basic_ack(delivery_tag = method.delivery_tag)channel.basic_qos(prefetch_count=1)channel.basic_consume(callback,                      queue='task_queue')channel.start_consuming()