爬虫笔记
来源:互联网 发布:精准医学大数据 编辑:程序博客网 时间:2024/05/22 21:02
爬虫
finder
request
urllib.Request(url,headers=)
—> 类似url post
用data传, get
拼接url 注意urllib.urlencode
(解码unquote
) request.add_header()
, request.get_full_url()
, request.get_host()
,request.get_header()
responce.code
,responce.info()
代理服务器
httpproxy = urllib2.ProxyHandler({'http':"账号:密码@主机:端口"})opener = urllib2.build_opener(httpproxy)responce = opener.open(url)responce.read()
下载
urllib.urlretrieve(url,file)
重定向
urllib2.HTTPRedirectHandler()urllib2.urlopen(url,timeout=)
代理服务器
pcUserAgent = {"safari 5.1 – MAC":"User-Agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50","safari 5.1 – Windows":"User-Agent:Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50","IE 9.0":"User-Agent:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0);","IE 8.0":"User-Agent:Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)","IE 7.0":"User-Agent:Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)","IE 6.0":"User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)","Firefox 4.0.1 – MAC":"User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Firefox 4.0.1 – Windows":"User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Opera 11.11 – MAC":"User-Agent:Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11","Opera 11.11 – Windows":"User-Agent:Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11","Chrome 17.0 – MAC":"User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11","Maxthon":"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)","Tencent TT":"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)","The World 2.x":"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)","The World 3.x":"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)","sogou 1.x":"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)","360":"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)","Avant":"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser)","Green Browser":"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)"}mobileUserAgent = {"iOS 4.33 – iPhone":"User-Agent:Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5","iOS 4.33 – iPod Touch":"User-Agent:Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5","iOS 4.33 – iPad":"User-Agent:Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5","Android N1":"User-Agent: Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1","Android QQ":"User-Agent: MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1","Android Opera ":"User-Agent: Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10","Android Pad Moto Xoom":"User-Agent: Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13","BlackBerry":"User-Agent: Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.337 Mobile Safari/534.1+","WebOS HP Touchpad":"User-Agent: Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.0; U; en-US) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/233.70 Safari/534.6 TouchPad/1.0","Nokia N97":"User-Agent: Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/20.0.019; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebKit/525 (KHTML, like Gecko) BrowserNG/7.1.18124","Windows Phone Mango":"User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan)","UC":"User-Agent: UCWEB7.0.2.37/28/999","UC standard":"User-Agent: NOKIA5700/ UCWEB7.0.2.37/28/999","UCOpenwave":"User-Agent: Openwave/ UCWEB7.0.2.37/28/999","UC Opera":"User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999"}
创建服务器对象,打开HTTPS debug log
urllib2.HTTPSHandler(debuglevel=1)
创建全局opener
urllib2.install_opener(opener)
创建对象存储cookie
cookie=cookielib.CookieJar()
可以操作的cookie对象
cookie=cookielib.LWPCookieJar(filepath)
创建一个链接对象,使用cookie,可用该header创建opener
header=urllib2.HTTPCookieProcessor(cookie)
保存cookie
cookie.save(ignore_discard=True,ignore_expires=True)
load cookie
cookie.load(filepath,ignore_discard=True,ignore_expires=True)
https,用context忽略安全,传入urlopen
context=ssl._create_unverified_context()
get post
req=requests.get(url,cookies=,params=,headers=,proxy=) req.text.encode("utf-8")req.contentreq.encodingreq.urlreq.history
cookie是dict(=)
proxy也是字典
自动监测编码
chardet.detect(req.content)
设置编码
chardet.detect(r.content)['encoding']req=requests.post(url,data=,headers=)
提取数据的方法
正则表达式
bs4
content=urllib2.urlopen(url).read()soup=BeautifulSoup(content,'html.parser') #进行数据类型转换,方便处理的数据
提取标签
bsobj.find("div",{"id":"mw-content-text"}).get_text()
LXML
html=lxml.etree.HTML(html_str)# 对象res=lxml.etree.tostring(html)# 详细内容
阅读全文
0 0
- 【爬虫笔记】爬虫入门
- 爬虫笔记
- 爬虫笔记
- 爬虫笔记
- 爬虫笔记
- 爬虫笔记
- 爬虫笔记
- 爬虫笔记
- 动态网页爬虫笔记
- 爬虫学习笔记
- 爬虫学习笔记
- Python爬虫笔记一
- Scrapy爬虫框架笔记
- Scrapy爬虫笔记-未完成
- 爬虫第一章笔记
- [笔记] PhantomJS爬虫小记
- 爬虫学习笔记
- Linkedin 数据爬虫笔记
- HDU 3549 Flow Problem(最大流+EK邻接表||邻接矩阵||dinic算法模板比较)
- 工厂模式
- ASP.net初步了解
- Java WebService服务+WSDL 实现手机号码归属地查询
- 旅游归来
- 爬虫笔记
- Lightoj-1356 Prime Independence(质因子分解)(Hopcroft-Karp优化的最大匹配)
- 设计内部类的好处
- 网络流24题-20
- strlen、sizeof、length的区别和联系
- 堆排序
- MFC学习笔记-ListControl更改单元格显示
- String 和StringBuffer的区别
- opencv中矩阵按行或者按列合并