Requests使用入门

Requests优点

• 底层实现为urllib
• 方法在Python2和Python3中通用
• 自动对gzip等压缩方式进行解压

作用

• 发送网络请求，返回响应数据
• 中文参考文档

发送简单请求

• 基本常用方法

# -*- coding:utf-8 -*-import requestsurl = 'http://www.baidu.com'response = requests.get(url)response.textresponse.contentresponse.status_coderesponse.request.headersresponse.headersresponse.url

• .text 和 .content的区别
  
  • response.text
    
    • 类型：str
    • 解码类型： 根据HTTP 头部对响应的编码作出有根据的推测，推测的文本编码
    • 如何修改编码方式：response.encoding=”gbk”
  • response.content
    
    • 类型：bytes
    • 解码类型： 没有指定
    • 如何修改编码方式：response.content.decode(“utf8”)
      一般使用response.content.decode()来获取网页内容

发送带header的请求

# coding=utf-8# demoimport requestsurl = 'http://www.baidu.com'header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.15 Safari/537.36'}response = requests.get(url, header=header)response.content.decode('gbk')

发送带参数请求

kw = dict(a:'50')response = requests.get(url, params = kw)

小案例：实现爬取任意网站内容并保存

# -*- coding:utf-8 -*-# thinking：1. 获取url# 2. 获取html字符串页面# 3. 保存html页面import requestsclass TieBaSpider(object):    def __init__(self, search_name):        self.search_name = search_name        temp_url = 'http://tieba.baidu.com/f?kw=("+self.search_name+")&ie=utf-8&pn={}'        self.tie_ba_url = []        for i in range(1000):            self.tie_ba_url.append(temp_url.format(i*50))        # 定义请求头        self.header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.15 Safari/537.36'}    def parse_url(self, url):        response = requests.get(url, params=self.header)        return response.content.decode('gbk')    # 获取html字符串    def save_html(self, html, page_num):        # 设置文件路径        file_path = self.search_name + '_' + str(page_num) + '.html'        with open(file_path, 'w') as f:            f.write(html)        print('保存成功')    def run(self):        # 取出url，获取html内容        for url in self.tie_ba_url:            html = self.parse_url(url)            # 获取页码            page_num = self.tie_ba_url.index(url)+1            self.save_html(html, page_num)if __name__ == "__main__":    tie_ba_name = input("请输入要爬取的贴吧名字：")    spride_test = TieBaSpider(tie_ba_name)    spride_test.run()