python urllib模块
来源:互联网 发布:华为云计算开发工程师 编辑:程序博客网 时间:2024/05/29 07:19
urllib提供了一系列用于操作URL的功能。
Get
urllib的request模块可以非常方便地抓取URL内容,也就是发送一个GET请求到指定的页面,然后返回HTTP的响应:方法是用urlopen函数,它的参数是url字符串或者是Request对象,他返回一个HTTPResponse对象
例如,对豆瓣的一个URLhttps://api.douban.com/v2/book/2129650进行抓取,并返回响应:
from urllib import requesturl='https://api.douban.com/v2/book/2129650'#urlopen的参数是url字符串或者是Request对象,返回值为HTTPResponsewith request.urlopen(url) as f: data=f.read() print('Statue: ',f.status,f.reason) for k,v in f.getheaders(): print('%s: %s' % (k,v)) print('Data: ',data.decode('utf-8'))
下面是HTTPResponse对象:
An HTTPResponse instance wraps the HTTP response from the server. It provides access to the request headers and the entity body. The response is an iterable object and can be used in a with statement.
HTTPResponse.read([amt])
Reads and returns the response body, or up to the next amt bytes.
HTTPResponse.readinto(b)
Reads up to the next len(b) bytes of the response body into the buffer b. Returns the number of bytes read.New in version 3.3.
HTTPResponse.getheader(name, default=None)
Return the value of the header name, or default if there is no header matching name. If there is more than one header with the name name, return all of the values joined by ‘, ‘. If ‘default’ is any iterable other than a single string, its elements are similarly returned joined by commas.
HTTPResponse.getheaders()
Return a list of (header, value) tuples.
HTTPResponse.fileno()
Return the fileno of the underlying socket.
HTTPResponse.msg
A http.client.HTTPMessage instance containing the response headers. http.client.HTTPMessage is a subclass of email.message.Message.
HTTPResponse.version
HTTP protocol version used by server. 10 for HTTP/1.0, 11 for HTTP/1.1.
HTTPResponse.status
Status code returned by server.
HTTPResponse.reason
Reason phrase returned by server.
HTTPResponse.debuglevel
A debugging hook. If debuglevel is greater than zero, messages will be printed to stdout as the response is read and parsed.
HTTPResponse.closed
Is True if the stream is closed.
如果我们要想模拟浏览器发送GET请求,就需要使用Request对象,通过往Request对象添加HTTP头,我们就可以把请求伪装成浏览器。例如,模拟火狐去请求Python首页:
其中User-agent是表示浏览器
Request对象都有什么属性和方法
from urllib import requesturl='https://www.python.org/'req=request.Request(url)req.add_header('User_agent','Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11')with request.urlopen(req) as f: print('Status:', f.status, f.reason) for k, v in f.getheaders(): print('%s: %s' % (k, v)) print('Data:', f.read().decode('utf-8'))
Get模拟微博登录:
from urllib import request,parseprint('Login to weibo.cn...')url='https://passport.weibo.cn/sso/login?username=xxxxxx&password=xxxxxx'print(url)req=request.Request(url)req.add_header('Origin', 'https://passport.weibo.cn')req.add_header('User-Agent', 'Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25')req.add_header('Referer', 'https://passport.weibo.cn/signin/login?entry=mweibo&res=wel&wm=3349&r=http%3A%2F%2Fm.weibo.cn%2F')with request.urlopen(req) as f: print('Status:', f.status, f.reason) for k, v in f.getheaders(): print('%s: %s' % (k, v)) print('Data:', f.read().decode('utf-8'))
Post
如果要以POST发送一个请求,只需要把参数data以bytes形式传入。
我们模拟一个微博登录,先读取登录的邮箱和口令,然后按照weibo.cn的登录页的格式以username=xxx&password=xxx的编码传入:
from urllib import request,parseprint('Login to weibo.cn...')url='https://passport.weibo.cn/sso/login'email=input('Email: ')password=input('Password: ')login_data=parse.urlencode([ ('username',email), ('password',password), ('entry','mweibo'), ('client_id',''), ('savestate','1'), ('ec',''), ('pagerefer', 'https://passport.weibo.cn/signin/welcome?entry=mweibo&r=http%3A%2F%2Fm.weibo.cn%2F')])req=request.Request(url)req.add_header('Origin', 'https://passport.weibo.cn')req.add_header('User-Agent', 'Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25')req.add_header('Referer', 'https://passport.weibo.cn/signin/login?entry=mweibo&res=wel&wm=3349&r=http%3A%2F%2Fm.weibo.cn%2F')with request.urlopen(req,data=login_data.encode('utf-8')) as f: print('Status:', f.status, f.reason) for k, v in f.getheaders(): print('%s: %s' % (k, v)) print('Data:', f.read().decode('utf-8'))
浅析HTTP协议
HTTP 请求方式: GET和POST的比较
http(百度百科)
HTTP协议详解
- python中的urllib模块
- Python模块学习 --- urllib
- Python模块学习 --- urllib
- Python模块学习 --- urllib
- Python模块学习 ---- urllib
- Python urllib模块学习
- Python模块学习 --- urllib
- Python模块学习 --- urllib
- Python urllib模块
- Python模块学习 --- urllib
- Python模块学习 --- urllib
- Python模块学习 --- urllib
- Python模块学习 --- urllib
- python模块学习 --- urllib
- python模块--urllib
- Python模块学习 --- urllib
- Python模块学习 --- urllib
- python urllib模块详解
- 堆排序
- linux下的tmpfs 文件系统
- 基于mmap的KV数据存储实现(一)概述
- ActiveMQ 中的消息游标(Message cursors)
- python之转换源码后缀名为txt后缀名
- python urllib模块
- centos启动php-fpm不使用默认9000端口Permission denied
- iOS开发:本地通知(推送)
- 关于Vim
- [面试算法题]比较二叉树异同-leetcode学习之旅(5)
- android SQLiteOpenHelper的使用方法
- 接口
- 使用Qt正则表达式提取全路径的文件名
- 15-11-18平静的一天