消息队列学习

在做微信开发的时候，要求服务器响应消息的时间在5秒内回复，如里没有达到5秒时间要求，会连续重发三次，都没有回复，将显示·「该公众号目前无法提供服务」。

在一些耗时的操作无法满足上面的条件。本来是不打算管的，结果连续报警两天，累觉不爱了。

解决这个问题就用到了消息队列机制，个人理解消息队列：就是把目前耗时的操作入队列存起来，先给用户回复，然后再从队列里读出耗时操作慢慢执行。

使用的是RabbitMQ消息队列，其官方地址：http://www.rabbitmq.com/tutorials/tutorial-three-python.html

下面是步骤 ：

一、安装 rabbitmq服务

sudo apt-get install rabbitmq-server

二、用的是python 的pika 

sudo apt-get install python-pip git-core python-pik

三、参考以下代码 

1、方式一

Work Queues

先回复，把耗时工作轮流的分发给C1、C2。

发送端：

new_task.py

#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.queue_declare(queue='task_queue', durable=True)message = ' '.join(sys.argv[1:]) or "Hello World!"channel.basic_publish(exchange='',                      routing_key='task_queue',                      body=message,                      properties=pika.BasicProperties(                         delivery_mode = 2, # make message persistent                      ))print " [x] Sent %r" % (message,)connection.close()

运行多个Cn。

接受端

worker.py

#!/usr/bin/env pythonimport pikaimport timeconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.queue_declare(queue='task_queue', durable=True)print ' [*] Waiting for messages. To exit press CTRL+C'def callback(ch, method, properties, body):    print " [x] Received %r" % (body,)    time.sleep( body.count('.') )    print " [x] Done"    ch.basic_ack(delivery_tag = method.delivery_tag)channel.basic_qos(prefetch_count=1)channel.basic_consume(callback,                      queue='task_queue')channel.start_consuming()

结果 ：

shell1$ python worker.py [*] Waiting for messages. To exit press CTRL+C

shell2$ python worker.py [*] Waiting for messages. To exit press CTRL+C

shell3$ python new_task.py First message.shell3$ python new_task.py Second message..shell3$ python new_task.py Third message...shell3$ python new_task.py Fourth message....shell3$ python new_task.py Fifth message.....

shell1$ python worker.py [*] Waiting for messages. To exit press CTRL+C [x] Received 'First message.' [x] Received 'Third message...' [x] Received 'Fifth message.....'

shell2$ python worker.py [*] Waiting for messages. To exit press CTRL+C [x] Received 'Second message..' [x] Received 'Fourth message....'

2、方式二

Publish/Subscribe

P广播消息给每一个C。

在这种方式我们创建了一个Exchanges，其类型是

type='fanout'

发送端代码

emit_log.py

#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.exchange_declare(exchange='logs',                         type='fanout')message = ' '.join(sys.argv[1:]) or "info: Hello World!"channel.basic_publish(exchange='logs',                      routing_key='',                      body=message)print " [x] Sent %r" % (message,)connection.close()

接收端代码

receive_logs.py

#!/usr/bin/env pythonimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.exchange_declare(exchange='logs',                         type='fanout')result = channel.queue_declare(exclusive=True)queue_name = result.method.queuechannel.queue_bind(exchange='logs',                   queue=queue_name)print ' [*] Waiting for logs. To exit press CTRL+C'def callback(ch, method, properties, body):    print " [x] %r" % (body,)channel.basic_consume(callback,                      queue=queue_name,                      no_ack=True)channel.start_consuming()

3、方式三

Routing

这种方式的Exchange的类型为direct。可以根据

routing_key=severity设定的值来过滤消息。比如C1只接受error类的消息。

发送端

emit_log_direct.py

#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.exchange_declare(exchange='direct_logs',                         type='direct')severity = sys.argv[1] if len(sys.argv) > 1 else 'info'message = ' '.join(sys.argv[2:]) or 'Hello World!'channel.basic_publish(exchange='direct_logs',                      routing_key=severity,                      body=message)print " [x] Sent %r:%r" % (severity, message)connection.close()

接受端：

receive_logs_direct.py

#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.exchange_declare(exchange='direct_logs',                         type='direct')result = channel.queue_declare(exclusive=True)queue_name = result.method.queueseverities = sys.argv[1:]if not severities:    print >> sys.stderr, "Usage: %s [info] [warning] [error]" % \                         (sys.argv[0],)    sys.exit(1)for severity in severities:    channel.queue_bind(exchange='direct_logs',                       queue=queue_name,                       routing_key=severity)print ' [*] Waiting for logs. To exit press CTRL+C'def callback(ch, method, properties, body):    print " [x] %r:%r" % (method.routing_key, body,)channel.basic_consume(callback,                      queue=queue_name,                      no_ack=True)channel.start_consuming()

4、方式四

Topics

这种方式的Exchange的类型为topics。可以根据

routing_key=severity 多个条件进行过滤。这里的routing_key形式是以.分隔连接的关键字字符串。比如：stock.usd.nyse。最长不能超过255字节。在过滤时， 可以类试正则表达式的匹配。

• * (star) can substitute for exactly one word.（代表一个关键字）
• # (hash) can substitute for zero or more words.（可表示0个，多个关键字）

发送端

emit_log_topic.py

#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.exchange_declare(exchange='topic_logs',                         type='topic')routing_key = sys.argv[1] if len(sys.argv) > 1 else 'anonymous.info'message = ' '.join(sys.argv[2:]) or 'Hello World!'channel.basic_publish(exchange='topic_logs',                      routing_key=routing_key,                      body=message)print " [x] Sent %r:%r" % (routing_key, message)connection.close() 

接收端

receive_logs_topic.py

#!/usr/bin/env pythonimport pikaimport sysconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.exchange_declare(exchange='topic_logs',                         type='topic')result = channel.queue_declare(exclusive=True)queue_name = result.method.queuebinding_keys = sys.argv[1:]if not binding_keys:    print >> sys.stderr, "Usage: %s [binding_key]..." % (sys.argv[0],)    sys.exit(1)for binding_key in binding_keys:    channel.queue_bind(exchange='topic_logs',                       queue=queue_name,                       routing_key=binding_key)print ' [*] Waiting for logs. To exit press CTRL+C'def callback(ch, method, properties, body):    print " [x] %r:%r" % (method.routing_key, body,)channel.basic_consume(callback,                      queue=queue_name,                      no_ack=True)channel.start_consuming()

运行结果

python receive_logs_topic.py "#"

python receive_logs_topic.py "kern.*"

python receive_logs_topic.py "kern.*" "*.critical"

python emit_log_topic.py "kern.critical" "A critical kernel error"

5、方式五

Remote procedure call (RPC)

远程过程调用，下面代码是关于在远程服务器上计算斐波那契数并返回。

客户端：

rpc_client.py

#!/usr/bin/env pythonimport pikaimport uuidclass FibonacciRpcClient(object):    def __init__(self):        self.connection = pika.BlockingConnection(pika.ConnectionParameters(                host='localhost'))        self.channel = self.connection.channel()        result = self.channel.queue_declare(exclusive=True)        self.callback_queue = result.method.queue        self.channel.basic_consume(self.on_response, no_ack=True,                                   queue=self.callback_queue)    def on_response(self, ch, method, props, body):        if self.corr_id == props.correlation_id:            self.response = body    def call(self, n):        self.response = None        self.corr_id = str(uuid.uuid4())        self.channel.basic_publish(exchange='',                                   routing_key='rpc_queue',                                   properties=pika.BasicProperties(                                         reply_to = self.callback_queue,                                         correlation_id = self.corr_id,                                         ),                                   body=str(n))        while self.response is None:            self.connection.process_data_events()        return int(self.response)fibonacci_rpc = FibonacciRpcClient()print " [x] Requesting fib(30)"response = fibonacci_rpc.call(30)print " [.] Got %r" % (response,)

服务端

rpc_server.py

#!/usr/bin/env pythonimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(        host='localhost'))channel = connection.channel()channel.queue_declare(queue='rpc_queue')def fib(n):    if n == 0:        return 0    elif n == 1:        return 1    else:        return fib(n-1) + fib(n-2)def on_request(ch, method, props, body):    n = int(body)    print " [.] fib(%s)"  % (n,)    response = fib(n)    ch.basic_publish(exchange='',                     routing_key=props.reply_to,                     properties=pika.BasicProperties(correlation_id = \                                                     props.correlation_id),                     body=str(response))    ch.basic_ack(delivery_tag = method.delivery_tag)channel.basic_qos(prefetch_count=1)channel.basic_consume(on_request, queue='rpc_queue')print " [x] Awaiting RPC requests"channel.start_consuming()

四、总结

RabbitMQ队列主的有三种方式

1、消息队列：即第一种工作方式，先返回消息后，慢慢处理耗时操作。轮流分发给多个worker

2、广播消息给多个worker。根据Exchange的类型（direct, topic, headers, fanout）及routing_key可以过滤消息。

3、远程过程调用