python3爬虫(三) BeautifulSoup模块的安装和介绍
来源:互联网 发布:web原型设计工具 知乎 编辑:程序博客网 时间:2024/05/22 03:05
个人博客:xiaogeng.top
前面我们介绍了requests库,简单入门了re正则表达式的基本用法,这次要介绍的是Beautiful Soup库,这是一个用于解析html或者xml文档的库,是解析、遍历、维护标签树的功能库。
尽管re正则已经可以处理和提取绝大多数的文档,但是配合Beautiful Soup库可以给我们带来极大的方便。
一、Beautiful Soup库的安装
Beautiful Soup库的安装最简单是用pip安装
在cmd下输入以下命令即可进行安装。
pip install beautifulsoup4
如果遇到问题可以下载安装包手动安装。网址:
https://www.crummy.com/software/BeautifulSoup/
具体安装过程不再繁述。如有问题可百度安装教程。
安装完成后我们来检测一下:
输入以下命令不出错即安装成功。
from bs4 import BeautifulSoup
二、Beautiful Soup库的使用
1、Beautiful Soup库的引用:
Beautiful Soup库也叫beautifulsoup4或者bs4。约定引用方式如下:
from bs4 import BeautifulSoupimport bs4
可看出主要用的是BeautifulSoup类。
2、BeautifulSoup解析器
BeautifulSoup可以解析多种格式的文档.
可以被用作不同的解析器
如:
soup=BeautifulSoup('<html>data</html>','html.parser')
其中html.parser就是基于bs4的html解析。
3、BeautifulSoup类的基本元素
代码演示:
>>> from bs4 import BeautifulSoup>>> import requests>>> demo=requests.get('https://python123.io/ws/demo.html').text>>> print(demo)<html><head><title>This is a python demo page</title></head><body><p class="title"><b>The demo python introduces several python courses.</b></p><p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:<a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p></body></html>>>> soup=BeautifulSoup(demo,'html.parser')#获取Tag标签内容>>> soup.title <title>This is a python demo page</title>>>> soup.a<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a># 获取Tag标签名>>> soup.a.name'a'>>> soup.p.name'p'>>> soup.a.parent.name'p'>>> soup.a.parent.parent.name'body'#获取Tag的attrs(属性),返回字典>>> soup.a.attrs{'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}>>> soup.a.attrs['class']['py1']#获取Tag的NavigableString(无属性字符串)>>> soup.a.string'Basic Python'#获取Tag内的注释部分 >>> newsoup=BeautifulSoup('<b><!-this is a comment--></b><p>this is content</p>','html.parser')>>> newsoup.b.string'this is a comment'#comment和NavigableString的类型不同!>>> type(newsoup.b.string)<class 'bs4.element.Comment'>>>> type(newsoup.p.string)<class 'bs4.element.NavigableString'>
4、BeautifulSoup的内容遍历方法
对于文档的标签遍历可分为下行遍历、上行遍历和平行遍历。
先来了解一下怎么找到一个标签的子标签和父标签和兄弟标签
- 下行遍历
代码举例:
>>> from bs4 import BeautifulSoup>>> import requests>>> demo=requests.get('https://python123.io/ws/demo.html')>>> demo<Response [200]>>>> demo=requests.get('https://python123.io/ws/demo.html').text>>> print(demo)<html><head><title>This is a python demo page</title></head><body><p class="title"><b>The demo python introduces several python courses.</b></p><p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:<a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p></body></html>#contents返回子节点的列表>>> print(soup.body.contents)['\n', <p class="title"><b>The demo python introduces several python courses.</b></p>, '\n', <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>, '\n']>>> len(soup.body.contents)5>>> print(soup.body.contents[1])<p class="title"><b>The demo python introduces several python courses.</b></p>
一般我们标签树的下行遍历是:
for child in soup.body.children:
print(child)for child in soup..body.descendants:
print(child)
- 上行遍历
#父亲标签>>> soup.title.parent<head><title>This is a python demo page</title></head>#遍历先辈标签>>> for parent in soup.a.parents: if parent is None: print(parent) else: print(parent.name)pbodyhtml[document]
- 平行遍历
迭代一般语法:
for sibling in soup.a.next_sibling: print(sibling)for sibling in soup.a.previous_sibling: print(sibling)
具体用法同上基本类似。
5、bs库的prettify()方法
能否让html内容更加友好的显示。这里有一个可以处理html显示格式的方法
.prettify()为html文本<>及其内容增加’\n’
.prettify()可用于标签,方法:< tag >.prettify()
我们来看一下具体效果
>>> soup<html><head><title>This is a python demo page</title></head><body><p class="title"><b>The demo python introduces several python courses.</b></p><p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p></body></html># >>> soup.prettify()'<html>\n <head>\n <title>\n This is a python demo page\n </title>\n </head>\n <body>\n <p class="title">\n <b>\n The demo python introduces several python courses.\n </b>\n </p>\n <p class="course">\n Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:\n <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">\n Basic Python\n </a>\n and\n <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">\n Advanced Python\n </a>\n .\n </p>\n </body>\n</html>'>>> print(soup.prettify())<html> <head> <title> This is a python demo page </title> </head> <body> <p class="title"> <b> The demo python introduces several python courses. </b> </p> <p class="course"> Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses: <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1"> Basic Python </a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2"> Advanced Python </a> . </p> </body></html>
怎么样是不是好看多了。
三、基于bs4库的html内容查找方法
这里主要介绍find_all方法,该方法返回一个列表类型,存储查找的结果。
具体格式如下:
<>.find_all ( name, attrs, recursive, string, **kwargs)
- name: 对标签名称的检索字符串
- attrs:对标签属性值的检索字符串,可标注属性检索。
- recursive:是否对子孙全部检索,默认为True
- string:< >…< /> 中字符串区域的检索字符串
- **kwargs :控制参数
先上代码就很清楚明了了
#1、name 对a标签名称的检索字符串。返回列表>>> soup.find_all('a')[<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>, <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>]# 同时检索a,b标签>>> soup.find_all(['a','b'])[<b>The demo python introduces several python courses.</b>, <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>, <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>]#2、标注了属性course>>> soup.find_all('p','course')[<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>]#3、调整recursive为False>>> soup.find_all('a',recursive=False)[]#4、设置string,寻找内容为string的字符串,返回列表>>> soup.find_all(string='Advanced Python')['Advanced Python']>>> import re>>> soup.find_all(string=re.compile('python'))['This is a python demo page', 'The demo python introduces several python courses.']
这里有一个简便一点的方法
1. < tag >(…)等价于< tag >.find_all(…)
2. soup()等价于soup.find_all(…)
最后总结一下除了find_all之外还有其他的扩展方法:
到这里我们就可以对大多数基本网页的爬取了。
- python3爬虫(三) BeautifulSoup模块的安装和介绍
- beautifulsoup介绍和安装
- python3 爬虫环境准备 (三)-常用库的安装
- python爬虫——windows + python3.4.3下的BeautifulSoup安装
- 爬虫----BeautifulSoup 模块
- python BeautifulSoup模块的安装
- pyhon BeautifulSoup模块的安装
- python BeautifulSoup模块的安装
- python BeautifulSoup模块的安装
- python3安装beautifulsoup包的方法
- 安装Python的easy_install工具和BeautifulSoup模块
- Python3.4 安装第三方模块Request、BeautifulSoup
- 网络爬虫之BeautifulSoup入门(三)
- Python爬虫入门(三)BeautifulSoup库
- Python3 win7安装BeautifulSoup
- Python3 win7安装BeautifulSoup
- python3.5安装BeautifulSoup
- BeautifulSoup的安装及介绍
- 三分法求单峰函数极值
- 谁能告诉我怎么获得c币啊
- MyBatis3 用log4j在控制台输出 SQL
- ListView限制最大显示高度笔记
- JavaScript中的跨域总结
- python3爬虫(三) BeautifulSoup模块的安装和介绍
- RadioGroup组件拖到按钮上方出现一个点而不是方框,
- 2017哈理工低年级组院赛决赛 I-幸运大奖
- 100、101、104-递归
- 用 C 语言画科赫雪花
- LeetCode —— 14. Longest Common Prefix解题思路
- MyBatis批量处理Oracle数据库数据
- Java面试题全集(下)
- Longest Valid Parentheses:最长括号子段匹配