python的requests初步使用

http://www.yangyanxing.com/?p=1079

早就听说requests的库的强大，只是还没有接触，今天接触了一下，发现以前使用urllib，urllib2等方法真是太搓了……

这里写些简单的使用初步作为一个记录

一、安装 http://cn.python-requests.org/en/latest/user/install.html#install

二、发送无参数的get请求

01

>>> r = requests.get('http://httpbin.org/get')

02

>>> print r.text

03

{

04

  "args": {},

05

  "headers": {

06

    "Accept":"*/*",

07

    "Accept-Encoding":"gzip, deflate",

08

    "Connection":"close",

09

    "Host":"httpbin.org",

10

    "User-Agent":"python-requests/2.3.0 CPython/2.6.6 Windows/7",

11

    "X-Request-Id":"8a28bbea-55cd-460b-bda3-f3427d66b700"

12

  },

13

  "origin":"124.192.129.84",

14

  "url":"http://httpbin.org/get"

15

}

Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com

三、发送带参数的get请求,将key与value放入一个字典中，通过params参数来传递,其作用相当于urllib.urlencode

1

>>> import requests

2

>>> pqyload = {'q':'杨彦星'}

3

>>> r = requests.get('http://www.so.com/s',params= pqyload)

4

>>> r.url

5

u'http://www.so.com/s?q=%E6%9D%A8%E5%BD%A6%E6%98%9F'

Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com

四、发送post请求，通过data参数来传递,

01

>>> payload = {'a':'杨','b':'hello'}

02

>>> r = requests.post("http://httpbin.org/post", data=payload)

03

>>> print r.text

04

{

05

  "args": {},

06

  "data": "",

07

  "files": {},

08

  "form": {

09

    "a":"u6768",

10

    "b":"hello"

11

  },

12

  "headers": {

13

    "Accept":"*/*",

14

    "Accept-Encoding":"gzip, deflate",

15

    "Connection":"close",

16

    "Content-Length":"19",

17

    "Content-Type":"application/x-www-form-urlencoded",

18

    "Host":"httpbin.org",

19

    "User-Agent":"python-requests/2.3.0 CPython/2.6.6 Windows/7",

20

    "X-Request-Id":"c81cb937-04b8-4a2d-ba32-04b5c0b3ba98"

21

  },

22

  "json": null,

23

  "origin":"124.192.129.84",

24

  "url":"http://httpbin.org/post"

25

}

26

>>>

Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com

可以看到，post参数已经传到了form里,data不光可以接受字典类型的数据，还可以接受json等格式

1

>>> payload = {'a':'杨','b':'hello'}

2

>>> import json

3

>>> r = requests.post('http://httpbin.org/post', data=json.dumps(payload))

Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com

五、发送文件的post类型，这个相当于向网站上传一张图片，文档等操作，这时要使用files参数

1

>>> url ='http://httpbin.org/post'

2

>>> files = {'file':open('touxiang.png','rb')}

3

>>> r = requests.post(url, files=files)

Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com

5.1 定制headers，使用headers参数来传递

1

>>> import json

2

>>> url ='https://api.github.com/some/endpoint'

3

>>> payload = {'some':'data'}

4

>>> headers = {'content-type':'application/json'}

5

 

6

>>> r = requests.post(url, data=json.dumps(payload), headers=headers)

Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com

 

六、响应内容

6.1 响应状态码

r = requests.get('http://httpbin.org/get')

print r.status_code

6.2 响应头

1

>>> print r.headers

2

{'content-length':'519', 'server':'gunicorn/18.0', 'connection':'keep-alive', 'date':'Sun, 15 Jun 2014 14:19:52 GMT', 'access-control-allow-origin': '*','content-type': 'application/json'}

Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com

也可以取到这个个别的响应头用来做一些判断，这里的参数是不区分大小写的

r.headers[‘Content-Type’]

r.headers.get(‘Content-Type’)

6.3 响应内容，前面已经在应用了

r.text

r.content

 

七、获取响应中的cookies

1

>>> r = requests.get('http://www.baidu.com')

2

>>> r.cookies['BAIDUID']

3

'D5810267346AEFB0F25CB0D6D0E043E6:FG=1'

Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com

也可以自已定义请求的COOKIES

01

>>> url ='http://httpbin.org/cookies'

02

>>> cookies = {'cookies_are':'working'}

03

>>> r = requests.get(url,cookies= cookies)

04

>>>

05

>>> print r.text

06

{

07

  "cookies": {

08

    "cookies_are":"working"

09

  }

10

}

11

>>>

Code inserted with Steve Dunn's Windows Live Writer Code Formatter Plugin. http://dunnhq.com

cookies还有很多，因为目前我也还不是很多，以后再扩充吧

八、使用timeout参数设置超时时间

>>> requests.get('http://github.com', timeout=1)

<Response [200]>

如果将时间设置成非常小的数，如requests.get('http://github.com', timeout=0.001)，那么如果在timeout的时间内没有连接，那么将会抛出一个Timeout的异常

九、访问中使用session

先初始化一个session对象，s = requests.Session()

然后使用这个session对象来进行访问，r = s.post(url,data = user)

 

参考文章 http://blog.csdn.net/iloveyin/article/details/21444613 基本上都是从这扒的代码

 

以下通过访问人人网来获取首页中的最近来访问，然后再访问查看更多的来访来读取更多的最近来访

更多的来访就是以带session的访问http://www.renren.com/myfoot.do

view source

01

#coding:utf-8

02

import requests

03

import re

04

 

05

url = r'http://www.renren.com/ajaxLogin'

06

 

07

user = {'email':'email','password':'pass'}

08

s = requests.Session()

09

r = s.post(url,data= user)

10

 

11

html = r.text

12

visit = []

13

first = re.compile(r'</span><span class="time-tip first-tip"><span class="tip-content">(.*?)</span>')

14

second = re.compile(r'</span><span class="time-tip"><span class="tip-content">(.*?)</span>')

15

third = re.compile(r'</span><span class="time-tip last-second-tip"><span class="tip-content">(.*?)</span>')

16

last = re.compile(r'</span><span class="time-tip last-tip"><span class="tip-content">(.*?)</span>')

17

visit.extend(first.findall(html))

18

visit.extend(second.findall(html))

19

visit.extend(third.findall(html))

20

visit.extend(last.findall(html))

21

for i in visit:

22

    print i

23

 

24

print '以下是更多的最近来访'

25

vm = s.get('http://www.renren.com/myfoot.do')

26

fm = re.compile(r'"name":"(.*?)"')

27

visitmore = fm.findall(vm.text)

28

for i in visitmore:

29

    print i