爬虫学习记录一

来源：互联网发布：辣鸭头淘宝配方编辑：程序博客网时间：2024/05/17 19:58

爬虫技术对于进行数据处理的相关研究者来说是非常重要的。要进行数据处理，首先我们必须要获取大量的数据，爬虫技术正好可以解决这个问题。对于计算机视觉方面的研究者来说，爬虫技术可以帮助我们快速构建数据集。

1.第一爬虫脚本如下：

from urllib2 import urlopenhtml = urlopen("http://pythonscraping.com/pages/page1.html")print(html.read())

python scrapy.py

结果：

<html><head><title>A Useful Page</title></head><body><h1>An Interesting Title</h1><div>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</div></body></html>

2.增加beatifulsoup

from urllib2 import urlopenfrom bs4 import BeautifulSouphtml = urlopen("http://www.pythonscraping.com/pages/page1.html")bsObj = BeautifulSoup(html.read())print(bsObj.h1)

结果：

<h1>An Interesting Title</h1>

考虑到url找不到等错误出现，修改了一下代码

from urllib2 import urlopenfrom bs4 import BeautifulSoupdef getTitle(url):    try:        html=urlopen(url)    except HTTPError as e:        return None    try:        bsObj=BeautifulSoup(html.read(),'lxml')        title=bsObj.body.h1    except AttributeError as e:        return None    else:        return titletitle=getTitle("http://pythonscraping.com/pages/page1.html")if title==None:    print("Title could not be found")else:    print(title)

结果

<h1>An Interesting Title</h1>

0 0