python爬虫之异步IO

在web请求非常多的情况下，异步IO往往效果和速率惊人。

#!/usr/bin/env python3# -*- coding: utf-8 -*-import requests,timestart_time=time.time()resp=requests.get(url='http://httpbin.org/headers')print(resp.text)print('用时：%s秒'%(time.time()-start_time))

运行结果：

然后用aiohttp和asyncio进行异步IO试一下：

#!/usr/bin/env python3# -*- coding: utf-8 -*-import asyncio, aiohttp, timestart_time=time.time()async def request(url):async with aiohttp.ClientSession() as session:async with session.get(url) as resp:resp_text=await resp.read()print(resp_text.decode('utf-8'))loop=asyncio.get_event_loop()tasks=[request('http://httpbin.org/headers')]loop.run_until_complete(asyncio.wait(tasks))print('用时：%s秒'%(time.time()-start_time))

运行结果：

虽然只有单任务，但是IO是通过交给操作系统处理的，速度也有一点的提升。那么在多任务的情况下会怎么样：

#!/usr/bin/env python3# -*- coding: utf-8 -*-import asyncio, aiohttp, timestart_time=time.time()response=[]async def fn(url):async with aiohttp.ClientSession() as session:async with session.get(url=url) as resp:response.append(await resp.read())loop=asyncio.get_event_loop()tasks=[asyncio.ensure_future(fn('http://www.baidu.com')) for x in range(500)]loop.run_until_complete(asyncio.wait(tasks))#print(response)print("用时%s秒" % (time.time()-start_time))

使用协程访问500次百度，运行结果：

我们再用同步的requests试一下：

60s!