BeautifulSoup Python抓网页小例子

来源：互联网发布：请大家悉知是什么意思编辑：程序博客网时间：2024/06/05 10:10

# -*- coding: utf-8 -*-import urllib2from BeautifulSoup import BeautifulSoup, Tagimport repage = urllib2.urlopen("http://bj.ganji.com/piao/zz_%E5%8C%97%E4%BA%AC-%E5%8D%97%E6%98%8C/20100210/")soup = BeautifulSoup(page)#ss = soup.findAll('a', href=re.compile(r"^/piao/100."))ss = soup.findAll(attrs={"class":"list_piao"})fp = open("c://Python25//web.html","w")doc = '''<html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /><title>keyunq tickets</title><meta http-equiv="refresh" content="5"/><link href="http://s1.ganjistatic1.com/css/base.__1265015655__.css" mce_href="http://s1.ganjistatic1.com/css/base.__1265015655__.css" rel="stylesheet" type="text/css" /><link href="http://s1.ganjistatic1.com/css/train.__1264669543__.css" mce_href="http://s1.ganjistatic1.com/css/train.__1264669543__.css" rel="stylesheet" type="text/css" /><mce:style></mce:style><style mce_bogus="1">.list_piao dt { float:left; width:40%; line-height:24px; font-size:14px; text-indent:5px;padding:5px 0;}</style></head><body>'''fp.write('%s/n' % doc)for i in ss: i.dt['class'] = 'list_piao_time' tmp = i.a['href'] i.a['href'] = 'http://bj.ganji.com'+tmp phonepage = urllib2.urlopen(i.a['href']) phonesoup = BeautifulSoup(phonepage) phone = phonesoup.findAll(attrs={"class":"phoneNum"}) tmp = phone[0].img['src'] phone[0].img['src'] = 'http://bj.ganji.com'+tmp tag1 = Tag(soup, "dd") tag1['class'] = 'list_piao_mj' i.insert(8,tag1) text = str(phone[0].img) tag1.insert(1,text) print i #i.dd.insert(0, str(phone[0].img)) fp.write('%s/n' % i)doc = '''</body></html>'''fp.write('%s/n' % doc)fp.close()

BeautifulSoup是用Python写的一个HTML/XML的解析器，它可以很好的处理不规范标记并生成剖析树(parse tree)。它提供简单又常用的导航（navigating），搜索以及修改剖析树的操作。它可以大大节省你的编程时间。

BeautifulSoup中文文档地址