Python 利用urllib2 lxml 抓取网页信息

来源：互联网发布：适合女生喝的红酒知乎编辑：程序博客网时间：2024/04/28 04:21

1. 本程序目的是提取网页中div class = "article_category" 的<a>标签下的text内容即黑体所示：

<div class="article_category"><a href="http://dir.yahoo.com/Recreation/Games/Video_Games/Titles/Recreation_and_Sports/Sports/Racing_and_Driving/MotorStorm" target="_blank">Recreation>Games>Video Games>Titles>Recreation and Sports>Sports>Racing and Driving>MotorStorm</a></div>

2. 使用的lib有：

import urllib2from lxml import etree

3. 源码：

（1）获取html内容：

def get_html(url):    try:        response = urllib2.urlopen(url,timeout = 30)        html = response.read()        return html    except Exception, e:        print e

（2）构建元素树etree，并利用xpath得到标签内容：

def get_cat(ht):    tree = etree.HTML(ht)    cont = tree.xpath("//div[@class='article_category']/a")    cat_ls = []    for x in cont:        cat_ls.append(x.text)                 return cat_ls

其中x.text记录了对应<a>标签内的文本内容

0 0