爬链家的房价数据

因为前阵子要买房子，所以自己写了个小的链家爬虫，分析一下房源价格位置啥的。爬虫也只有在想要用的时候，才有作用啊。漫无目的的爬，总觉得提不起兴致。

浏览器的查询xpath的工具是firebug，用firefox查起来比较方便。

链家有些信息做得还是很粗糙的，所以没法快速结构化，索性爬下来后用excel处理一下就好了。比如下面的户型、面积，是揉捏在一块的，只用“|”分隔。为图方便，全部抓取下来后，用excel分裂处理即可。

源码：

# -*- coding: utf-8 -*-import scrapyfrom lianjia.items import lianjiafrom scrapy.http import Requestimport jsonimport reclass DmozSpider(scrapy.Spider):    name = "lianjia"    allowed_domains = ["sh.lianjia.com"]    start_urls = [        'http://sh.lianjia.com/ershoufang/beicai/b250to350',        'http://sh.lianjia.com/ershoufang/biyun/b250to350',        'http://sh.lianjia.com/ershoufang/caolu/b250to350',        'http://sh.lianjia.com/ershoufang/chuansha/b250to350',        'http://sh.lianjia.com/ershoufang/datuanzhen/b250to350',        'http://sh.lianjia.com/ershoufang/gaodong/b250to350',        'http://sh.lianjia.com/ershoufang/gaohang/b250to350',        'http://sh.lianjia.com/ershoufang/huamu/b250to350',        'http://sh.lianjia.com/ershoufang/huinan/b250to350',        'http://sh.lianjia.com/ershoufang/geqing/b250to350',        'http://sh.lianjia.com/ershoufang/hangtou/b250to350',        'http://sh.lianjia.com/ershoufang/jinqiao/b250to350',        'http://sh.lianjia.com/ershoufang/jinyang/b250to350',        'http://sh.lianjia.com/ershoufang/kangqiao/b250to350',        'http://sh.lianjia.com/ershoufang/lingangxincheng/b250to350',        'http://sh.lianjia.com/ershoufang/laogangzhen/b250to350',        'http://sh.lianjia.com/ershoufang/lujiazui/b250to350',        'http://sh.lianjia.com/ershoufang/lianyang/b250to350',        'http://sh.lianjia.com/ershoufang/nichengzhen/b250to350',        'http://sh.lianjia.com/ershoufang/nanmatou/b250to350',        'http://sh.lianjia.com/ershoufang/shibo/b250to350',        'http://sh.lianjia.com/ershoufang/sanlin/b250to350',        'http://sh.lianjia.com/ershoufang/shuyuanzhen/b250to350',        'http://sh.lianjia.com/ershoufang/tangqiao/b250to350',        'http://sh.lianjia.com/ershoufang/tangzhen/b250to350',        'http://sh.lianjia.com/ershoufang/weifang/b250to350',        'http://sh.lianjia.com/ershoufang/waigaoqiao/b250to350',        'http://sh.lianjia.com/ershoufang/wanxiangzhen/b250to350',        'http://sh.lianjia.com/ershoufang/xinchang/b250to350',        'http://sh.lianjia.com/ershoufang/xuanqiao/b250to350',        'http://sh.lianjia.com/ershoufang/yangdong/b250to350',        'http://sh.lianjia.com/ershoufang/yangjing/b250to350',        'http://sh.lianjia.com/ershoufang/yuqiao1/b250to350',        'http://sh.lianjia.com/ershoufang/yuanshen/b250to350',        'http://sh.lianjia.com/ershoufang/zhangjiang/b250to350',        'http://sh.lianjia.com/ershoufang/zhoupu/b250to350',        'http://sh.lianjia.com/ershoufang/zhuqiao/b250to350',        'http://sh.lianjia.com/ershoufang/chunshen/b250to350',        'http://sh.lianjia.com/ershoufang/gumei/b250to350',        'http://sh.lianjia.com/ershoufang/huacao/b250to350',        'http://sh.lianjia.com/ershoufang/hanghua/b250to350',        'http://sh.lianjia.com/ershoufang/jinganxincheng/b250to350',        'http://sh.lianjia.com/ershoufang/jinhui/b250to350',        'http://sh.lianjia.com/ershoufang/jinhongqiao/b250to350',        'http://sh.lianjia.com/ershoufang/longbai/b250to350',        'http://sh.lianjia.com/ershoufang/laominhang/b250to350',        'http://sh.lianjia.com/ershoufang/meilong/b250to350',        'http://sh.lianjia.com/ershoufang/maqiao/b250to350',        'http://sh.lianjia.com/ershoufang/pujiang1/b250to350',        'http://sh.lianjia.com/ershoufang/qibao/b250to350',        'http://sh.lianjia.com/ershoufang/wujing/b250to350',        'http://sh.lianjia.com/ershoufang/shenzhuang/b250to350',        'http://sh.lianjia.com/ershoufang/zhuanqiao/b250to350',        'http://sh.lianjia.com/ershoufang/dachangzhen/b250to350',        'http://sh.lianjia.com/ershoufang/dahua/b250to350',        'http://sh.lianjia.com/ershoufang/gucun/b250to350',        'http://sh.lianjia.com/ershoufang/gongfu/b250to350',        'http://sh.lianjia.com/ershoufang/gaojing/b250to350',        'http://sh.lianjia.com/ershoufang/gongkang/b250to350',        'http://sh.lianjia.com/ershoufang/luodian/b250to350',        'http://sh.lianjia.com/ershoufang/luojing/b250to350',        'http://sh.lianjia.com/ershoufang/songbao/b250to350',        'http://sh.lianjia.com/ershoufang/shangda/b250to350',        'http://sh.lianjia.com/ershoufang/songnan/b250to350',        'http://sh.lianjia.com/ershoufang/tonghe/b250to350',        'http://sh.lianjia.com/ershoufang/yanghang/b250to350',        'http://sh.lianjia.com/ershoufang/yuepu/b250to350',        'http://sh.lianjia.com/ershoufang/zhangmiao/b250to350',        'http://sh.lianjia.com/ershoufang/caohejing/b250to350',        'http://sh.lianjia.com/ershoufang/changqiao/b250to350',        'http://sh.lianjia.com/ershoufang/huadongligong/b250to350',        'http://sh.lianjia.com/ershoufang/huajing/b250to350',        'http://sh.lianjia.com/ershoufang/hengshanlu/b250to350',        'http://sh.lianjia.com/ershoufang/jianguoxilu/b250to350',        'http://sh.lianjia.com/ershoufang/kangjian/b250to350',        'http://sh.lianjia.com/ershoufang/longhua/b250to350',        'http://sh.lianjia.com/ershoufang/shanghainanzhan/b250to350',        'http://sh.lianjia.com/ershoufang/tianlin/b250to350',        'http://sh.lianjia.com/ershoufang/wantiguan/b250to350',        'http://sh.lianjia.com/ershoufang/xuhuibinjiang/b250to350',        'http://sh.lianjia.com/ershoufang/xujiahui/b250to350',        'http://sh.lianjia.com/ershoufang/xietulu/b250to350',        'http://sh.lianjia.com/ershoufang/zhiwuyuan/b250to350',        'http://sh.lianjia.com/ershoufang/changfeng1/b250to350',        'http://sh.lianjia.com/ershoufang/changshoulu/b250to350',        'http://sh.lianjia.com/ershoufang/caoyang/b250to350',        'http://sh.lianjia.com/ershoufang/changzheng/b250to350',        'http://sh.lianjia.com/ershoufang/ganquanyichuan/b250to350',        'http://sh.lianjia.com/ershoufang/guangxin/b250to350',        'http://sh.lianjia.com/ershoufang/taopu/b250to350',        'http://sh.lianjia.com/ershoufang/wanli/b250to350',        'http://sh.lianjia.com/ershoufang/wuning/b250to350',        'http://sh.lianjia.com/ershoufang/zhenguang/b250to350',        'http://sh.lianjia.com/ershoufang/zhenru/b250to350',        'http://sh.lianjia.com/ershoufang/zhongyuanliangwancheng/b250to350',        'http://sh.lianjia.com/ershoufang/anshan/b250to350',        'http://sh.lianjia.com/ershoufang/dongwaitan/b250to350',        'http://sh.lianjia.com/ershoufang/huangxinggongyuan/b250to350',        'http://sh.lianjia.com/ershoufang/kongjianglu/b250to350',        'http://sh.lianjia.com/ershoufang/wujiaochang/b250to350',        'http://sh.lianjia.com/ershoufang/xinjiangwancheng/b250to350',        'http://sh.lianjia.com/ershoufang/zhoujiazuilu/b250to350',        'http://sh.lianjia.com/ershoufang/zhongyuan1/b250to350',        'http://sh.lianjia.com/ershoufang/beixinjing/b250to350',        'http://sh.lianjia.com/ershoufang/gubei/b250to350',        'http://sh.lianjia.com/ershoufang/hongqiao1/b250to350',        'http://sh.lianjia.com/ershoufang/tianshan/b250to350',        'http://sh.lianjia.com/ershoufang/xinhualu/b250to350',        'http://sh.lianjia.com/ershoufang/xijiao/b250to350',        'http://sh.lianjia.com/ershoufang/xianxia/b250to350',        'http://sh.lianjia.com/ershoufang/zhenninglu/b250to350',        'http://sh.lianjia.com/ershoufang/zhongshangongyuan/b250to350',        'http://sh.lianjia.com/ershoufang/chedun/b250to350',        'http://sh.lianjia.com/ershoufang/jiuting/b250to350',        'http://sh.lianjia.com/ershoufang/maogang/b250to350',        'http://sh.lianjia.com/ershoufang/shihudang/b250to350',        'http://sh.lianjia.com/ershoufang/sijing/b250to350',        'http://sh.lianjia.com/ershoufang/songjiangdaxuecheng/b250to350',        'http://sh.lianjia.com/ershoufang/songjianglaocheng/b250to350',        'http://sh.lianjia.com/ershoufang/songjiangxincheng/b250to350',        'http://sh.lianjia.com/ershoufang/sheshan/b250to350',        'http://sh.lianjia.com/ershoufang/xinbang/b250to350',        'http://sh.lianjia.com/ershoufang/xiaokunshan/b250to350',        'http://sh.lianjia.com/ershoufang/shenminbieshu/b250to350',        'http://sh.lianjia.com/ershoufang/xinqiao/b250to350',        'http://sh.lianjia.com/ershoufang/yexie/b250to350',        'http://sh.lianjia.com/ershoufang/anting/b250to350',        'http://sh.lianjia.com/ershoufang/fengzhuang/b250to350',        'http://sh.lianjia.com/ershoufang/huating/b250to350',        'http://sh.lianjia.com/ershoufang/jiadinglaocheng/b250to350',        'http://sh.lianjia.com/ershoufang/jiadingxincheng/b250to350',        'http://sh.lianjia.com/ershoufang/jiangqiao/b250to350',        'http://sh.lianjia.com/ershoufang/juyuanxinqu/b250to350',        'http://sh.lianjia.com/ershoufang/malu/b250to350',        'http://sh.lianjia.com/ershoufang/nanxiang/b250to350',        'http://sh.lianjia.com/ershoufang/waigang/b250to350',        'http://sh.lianjia.com/ershoufang/xinchenglu1/b250to350',        'http://sh.lianjia.com/ershoufang/xuxing/b250to350',        'http://sh.lianjia.com/ershoufang/dongjiadu/b250to350',        'http://sh.lianjia.com/ershoufang/dapuqiao/b250to350',        'http://sh.lianjia.com/ershoufang/huaihaizhonglu/b250to350',        'http://sh.lianjia.com/ershoufang/huangpubinjiang/b250to350',        'http://sh.lianjia.com/ershoufang/laoximen/b250to350',        'http://sh.lianjia.com/ershoufang/nanjingdonglu/b250to350',        'http://sh.lianjia.com/ershoufang/penglaigongyuan/b250to350',        'http://sh.lianjia.com/ershoufang/renminguangchang/b250to350',        'http://sh.lianjia.com/ershoufang/shibobinjiang/b250to350',        'http://sh.lianjia.com/ershoufang/wuliqiao/b250to350',        'http://sh.lianjia.com/ershoufang/xintiandi/b250to350',        'http://sh.lianjia.com/ershoufang/yuyuan/b250to350',        'http://sh.lianjia.com/ershoufang/caojiadu/b250to350',        'http://sh.lianjia.com/ershoufang/jingansi/b250to350',        'http://sh.lianjia.com/ershoufang/jiangninglu/b250to350',        'http://sh.lianjia.com/ershoufang/nanjingxilu/b250to350',        'http://sh.lianjia.com/ershoufang/buyecheng/b250to350',        'http://sh.lianjia.com/ershoufang/daning/b250to350',        'http://sh.lianjia.com/ershoufang/pengpu/b250to350',        'http://sh.lianjia.com/ershoufang/xizangbeilu/b250to350',        'http://sh.lianjia.com/ershoufang/yangcheng/b250to350',        'http://sh.lianjia.com/ershoufang/yonghe/b250to350',        'http://sh.lianjia.com/ershoufang/zhabeigongyuan/b250to350',        'http://sh.lianjia.com/ershoufang/beiwaitan/b250to350',        'http://sh.lianjia.com/ershoufang/jiangwanzhen/b250to350',        'http://sh.lianjia.com/ershoufang/liangcheng/b250to350',        'http://sh.lianjia.com/ershoufang/linpinglu/b250to350',        'http://sh.lianjia.com/ershoufang/luxungongyuan/b250to350',        'http://sh.lianjia.com/ershoufang/quyang/b250to350',        'http://sh.lianjia.com/ershoufang/sichuanbeilu/b250to350',        'http://sh.lianjia.com/ershoufang/baihe/b250to350',        'http://sh.lianjia.com/ershoufang/chonggu/b250to350',        'http://sh.lianjia.com/ershoufang/huaxin/b250to350',        'http://sh.lianjia.com/ershoufang/jinze/b250to350',        'http://sh.lianjia.com/ershoufang/liantang1/b250to350',        'http://sh.lianjia.com/ershoufang/xianghuaqiao/b250to350',        'http://sh.lianjia.com/ershoufang/xujing/b250to350',        'http://sh.lianjia.com/ershoufang/xiayang/b250to350',        'http://sh.lianjia.com/ershoufang/yingpu/b250to350',        'http://sh.lianjia.com/ershoufang/zhujiajiao/b250to350',        'http://sh.lianjia.com/ershoufang/zhaoxiang/b250to350',        'http://sh.lianjia.com/ershoufang/fengcheng/b250to350',        'http://sh.lianjia.com/ershoufang/fengxianjinhui/b250to350',        'http://sh.lianjia.com/ershoufang/haiwan/b250to350',        'http://sh.lianjia.com/ershoufang/nanqiao/b250to350',        'http://sh.lianjia.com/ershoufang/qingcun/b250to350',        'http://sh.lianjia.com/ershoufang/situan/b250to350',        'http://sh.lianjia.com/ershoufang/xidu/b250to350',        'http://sh.lianjia.com/ershoufang/zhuanghang/b250to350',        'http://sh.lianjia.com/ershoufang/zhelin/b250to350',        'http://sh.lianjia.com/ershoufang/caojing/b250to350',        'http://sh.lianjia.com/ershoufang/fengjing/b250to350',        'http://sh.lianjia.com/ershoufang/jinshan1/b250to350',        'http://sh.lianjia.com/ershoufang/langxia/b250to350',        'http://sh.lianjia.com/ershoufang/luxiang/b250to350',        'http://sh.lianjia.com/ershoufang/shihua/b250to350',        'http://sh.lianjia.com/ershoufang/shanyang/b250to350',        'http://sh.lianjia.com/ershoufang/tinglin/b250to350',        'http://sh.lianjia.com/ershoufang/zhujing/b250to350',        'http://sh.lianjia.com/ershoufang/zhangyan/b250to350',        'http://sh.lianjia.com/ershoufang/baozhen/b250to350',        'http://sh.lianjia.com/ershoufang/chenjiazhen/b250to350',        'http://sh.lianjia.com/ershoufang/chongmingqita/b250to350',        'http://sh.lianjia.com/ershoufang/chongmingxincheng/b250to350',        'http://sh.lianjia.com/ershoufang/changxingdao21211/b250to350',        'http://sh.lianjia.com/ershoufang/hengshadao/b250to350',        'http://sh.lianjia.com/ershoufang/haimen/b250to350',        'http://sh.lianjia.com/ershoufang/jiaxing/b250to350',        'http://sh.lianjia.com/ershoufang/kunshan1/b250to350',        'http://sh.lianjia.com/ershoufang/qidong1/b250to350',        'http://sh.lianjia.com/ershoufang/shanghaizhoubian2/b250to350',        'http://sh.lianjia.com/ershoufang/suzhou/b250to350',        'http://sh.lianjia.com/ershoufang/taicang212/b250to350',    ]    def parse(self,response):        item_house=response.xpath('//ul[@id="house-lst"]/li/div[@class="info-panel"]')        for i in item_house:            item=lianjia()            item['url'] = i.xpath('h2/a/@href').extract()            item['community'] = i.xpath('div[1]/div[1]/a/span/text()').extract()            item['room'] = i.xpath('div[1]/div[1]/span[1]/text()').extract()            item['proportion'] = i.xpath('div[1]/div[1]/span[2]/text()').extract()            item['otherInfo'] = i.xpath('div[1]/div[2]/div').extract()            item['price'] = i.xpath('div[2]/div[1]/span/text()').extract()            item['unit_price'] = i.xpath('div[2]/div[2]/text()').extract()            item['subway'] = i.xpath('div[1]/div[3]/div/div/span[2]/span/text()').extract()            item['tax_free'] = i.xpath('div[1]/div[3]/div/div/span[4]/span/text()').extract()            item['has_key'] = i.xpath('div[1]/div[3]/div/div/span[4]/span/text()').extract()            item['show_time'] = i.xpath('div[3]/div/div[1]/span/text()').extract()            yield item        next_page = response.xpath('//a[@gahref="results_next_page"]/@href').extract()        if next_page:            next_page = 'http://sh.lianjia.com' + next_page[0].encode("utf-8")            yield Request(next_page, callback=self.parse)

注1：

为啥start_urls那么长，因为不高兴写什么正则了，查到上海商圈的list，就直接拿来用。简单，粗暴。

注2：

b250to350

这是价格在250到350之间的意思，还有其他筛选条件，例如面积、户型等。