Python爬虫包 BeautifulSoup 学习(三) 实例
来源:互联网 发布:fanuc编程模拟软件 编辑:程序博客网 时间:2024/05/16 05:23
一步一步构建一个爬虫实例,抓取糗事百科的段子
先不用beautifulsoup包来进行解析
第一步,访问网址并抓取源码
# -*- coding: utf-8 -*-# @Author: HaonanWu# @Date: 2016-12-22 16:16:08# @Last Modified by: HaonanWu# @Last Modified time: 2016-12-22 20:17:13import urllibimport urllib2import reimport osif __name__ == '__main__': # 访问网址并抓取源码 url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357' user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} try: request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) content = response.read() except urllib2.HTTPError as e: print e exit() except urllib2.URLError as e: print e exit() print content.decode('utf-8')
第二步,利用正则表达式提取信息
首先先观察源码中,你需要的内容的位置以及如何识别
然后用正则表达式去识别读取
注意正则表达式中的 . 是不能匹配\n的,所以需要设置一下匹配模式。
# -*- coding: utf-8 -*-# @Author: HaonanWu# @Date: 2016-12-22 16:16:08# @Last Modified by: HaonanWu# @Last Modified time: 2016-12-22 20:17:13import urllibimport urllib2import reimport osif __name__ == '__main__': # 访问网址并抓取源码 url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357' user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} try: request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) content = response.read() except urllib2.HTTPError as e: print e exit() except urllib2.URLError as e: print e exit() regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S) items = re.findall(regex, content) # 提取数据 # 注意换行符,设置 . 能够匹配换行符 for item in items: print item
第三步,修正数据并保存到文件中
# -*- coding: utf-8 -*-# @Author: HaonanWu# @Date: 2016-12-22 16:16:08# @Last Modified by: HaonanWu# @Last Modified time: 2016-12-22 21:41:32import urllibimport urllib2import reimport osif __name__ == '__main__': # 访问网址并抓取源码 url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357' user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} try: request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) content = response.read() except urllib2.HTTPError as e: print e exit() except urllib2.URLError as e: print e exit() regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S) items = re.findall(regex, content) # 提取数据 # 注意换行符,设置 . 能够匹配换行符 path = './qiubai' if not os.path.exists(path): os.makedirs(path) count = 1 for item in items: #整理数据,去掉\n,将<br/>换成\n item = item.replace('\n', '').replace('<br/>', '\n') filepath = path + '/' + str(count) + '.txt' f = open(filepath, 'w') f.write(item) f.close() count += 1
第四步,将多个页面下的内容都抓取下来
# -*- coding: utf-8 -*-# @Author: HaonanWu# @Date: 2016-12-22 16:16:08# @Last Modified by: HaonanWu# @Last Modified time: 2016-12-22 20:17:13import urllibimport urllib2import reimport osif __name__ == '__main__': # 访问网址并抓取源码 path = './qiubai' if not os.path.exists(path): os.makedirs(path) user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S) count = 1 for cnt in range(1, 35): print '第' + str(cnt) + '轮' url = 'http://www.qiushibaike.com/textnew/page/' + str(cnt) + '/?s=4941357' try: request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) content = response.read() except urllib2.HTTPError as e: print e exit() except urllib2.URLError as e: print e exit() # print content # 提取数据 # 注意换行符,设置 . 能够匹配换行符 items = re.findall(regex, content) # 保存信息 for item in items: # print item #整理数据,去掉\n,将<br/>换成\n item = item.replace('\n', '').replace('<br/>', '\n') filepath = path + '/' + str(count) + '.txt' f = open(filepath, 'w') f.write(item) f.close() count += 1 print '完成'
使用BeautifulSoup对源码进行解析
# -*- coding: utf-8 -*-# @Author: HaonanWu# @Date: 2016-12-22 16:16:08# @Last Modified by: HaonanWu# @Last Modified time: 2016-12-22 21:34:02import urllibimport urllib2import reimport osfrom bs4 import BeautifulSoupif __name__ == '__main__': url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357' user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' headers = {'User-Agent':user_agent} request = urllib2.Request(url = url, headers = headers) response = urllib2.urlopen(request) # print response.read() soup_packetpage = BeautifulSoup(response, 'lxml') items = soup_packetpage.find_all("div", class_="content") for item in items: try: content = item.span.string except AttributeError as e: print e exit() if content: print content + "\n"
这是用BeautifulSoup去抓取书本以及其价格的代码
可以通过对比得出到bs4对标签的读取以及标签内容的读取
(因为我自己也没有学到这一部分,目前只能依葫芦画瓢地写)
# -*- coding: utf-8 -*-# @Author: HaonanWu# @Date: 2016-12-22 20:37:38# @Last Modified by: HaonanWu# @Last Modified time: 2016-12-22 21:27:30import urllib2import urllibimport re from bs4 import BeautifulSoup url = "https://www.packtpub.com/all"try: html = urllib2.urlopen(url) except urllib2.HTTPError as e: print e exit()soup_packtpage = BeautifulSoup(html, 'lxml') all_book_title = soup_packtpage.find_all("div", class_="book-block-title") price_regexp = re.compile(u"\s+\$\s\d+\.\d+") for book_title in all_book_title: try: print "Book's name is " + book_title.string.strip() except AttributeError as e: print e exit() book_price = book_title.find_next(text=price_regexp) try: print "Book's price is "+ book_price.strip() except AttributeError as e: print e exit() print ""
0 0
- Python爬虫包 BeautifulSoup 学习(三) 实例
- Python爬虫包 BeautifulSoup 学习(五) 实例
- Python爬虫包 BeautifulSoup 学习(一) 简介与安装
- Python爬虫包 BeautifulSoup 学习(二) 异常处理
- Python爬虫包 BeautifulSoup 学习(六) 递归抓取
- Python爬虫包 BeautifulSoup 学习(七) children等应用
- Python爬虫包 BeautifulSoup 学习(八) parent等应用
- Python爬虫包 BeautifulSoup 学习(十一) CSS 选择器
- Python爬虫入门(三)BeautifulSoup库
- python beautifulsoup 爬虫学习
- Python爬虫包 BeautifulSoup 学习(四) bs基本对象与函数
- Python爬虫包 BeautifulSoup 学习(九) 正则表达式与Lambda表达式
- Python爬虫包 BeautifulSoup 学习(十) 各种html解析器的比较及使用
- Python---BeautifulSoup 简单的爬虫实例
- python网络爬虫与信息采取之解析网页(三)---- BeautifulSoup库的导航树实例
- Python爬虫学习纪要(一):BeautifulSoup相关知识点
- Python爬虫学习纪要(二):BeautifulSoup相关知识点2
- Python爬虫学习纪要(十一):BeautifulSoup相关知识点3
- 12.玩转Spring Boot 事务管理
- erdaicms首款基于weui(微信网页开发样式库)的前端模版上线
- CVPR 2016-12-20
- 卷积神经网络反向传播推导
- VxWorks -- 信号量
- Python爬虫包 BeautifulSoup 学习(三) 实例
- sicily Message Flood STL中set和map的应用
- 二分搜索
- 设计模式-适配器模式
- boost学习之asio—chat
- 数据库系统概论 第1章--绪论--第2节-数据模型
- LVS-DR(Virtual Server via Direct Routing)原理说明与配置用例
- ServletConfig与ServletContext对象详解
- ruby:json