Scrapy简明教程(四)——爬取CSDN博客专家所有博文并存入MongoDB

  首先，我们来看一下CSDN博客专家的链接: http://blog.csdn.net/experts.html

上图为 CSDN 所有博客专家页面，点击下一页后发现每次 url 都不会改变，但是已经翻页了，检查网页元素如下图:

   我们发现可以通过 value 值来构造 url 实现翻页，&page=1代表第一页，先来看一下构造的 CSDN 博客专家首页: http://blog.csdn.net/peoplelist.html?channelid=0&page=1，没有了刚才炫酷的样式，但是并不影响我们爬取数据。

这里，我们随机访问一位博主的首页，然后点击下一页:

   可以发现 list 后面的数字就是页码，通过它可以构造博主所有文章翻页 url，然后爬取每页博文 url，最后爬取博文详情页信息。下面开始编写爬虫，创建项目请参考：Scrapy简明教程(二)——开启Scrapy爬虫项目之旅，下面直接讲编写其他组件:

1. 编写 settings.py :

设置用户代理，解除 ITEM_PIPELINES 注释，用户代理可在审查元素中查看:

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'ITEM_PIPELINES = {    'csdnblog.pipelines.CsdnblogPipeline': 300,}

2. 编写要抽取的数据域 (items.py) :

# -*- coding: utf-8 -*-# Define here the models for your scraped items## See documentation in:# http://doc.scrapy.org/en/latest/topics/items.htmlimport scrapyclass CsdnblogItem(scrapy.Item):    # define the fields for your item here like:    title = scrapy.Field()    url = scrapy.Field()    releaseTime = scrapy.Field()    readnum = scrapy.Field()    article = scrapy.Field()    keywords = scrapy.Field()

3. 编写 piplines.py:

# -*- coding: utf-8 -*-# Define your item pipelines heremZ## Don't forget to add your pipeline to the ITEM_PIPELINES setting# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.htmlimport sysreload(sys)sys.setdefaultencoding("utf-8")from pymongo import MongoClientclass CsdnblogPipeline(object):    def __init__(self):        self.client = MongoClient('localhost', 27017)        mdb = self.client['csdnblog']        self.collection = mdb['csdnblog']    def process_item(self, item, spider):        data = dict(item)        self.collection.insert(data)        return item    def close_spider(self, spider):        self.client.close()

4. 编写爬虫文件:

# -*- coding: utf-8 -*-import scrapyimport reimport urllib2import lxml.htmlimport mathimport jiebaimport jieba.analysefrom optparse import OptionParserfrom docopt import docoptfrom scrapy.http import Requestfrom csdnblog.items import CsdnblogItemimport sysreload(sys)sys.setdefaultencoding('utf-8')class SpiderCsdnblogSpider(scrapy.Spider):    name = "spider_csdnblog"    allowed_domains = ["csdn.net"]    start_urls = ['http://blog.csdn.net/peoplelist.html?channelid=0&page=1']    def parse(self, response):        # 从所有博客专家首页开始抓取        data = response.xpath('/html/body/div/span/text()').extract()[0]        # 抓取页码        pages = re.findall('共(.*?)页', str(data))[0]        for i in range(0, int(pages)):            # 构造博客专家翻页url            purl = 'http://blog.csdn.net/peoplelist.html?channelid=0&page=' + str(i+1)            yield Request(url=purl, callback=self.blogger)    def blogger(self, response):        # 爬取当前页博客专家url        bloggers = response.xpath('/html/body/dl/dd/a/@href').extract()        for burl in bloggers:            yield Request(url=burl, callback=self.total)    def total(self, response):        data = response.xpath('//*[@id="papelist"]/span/text()').extract()[0]        pages = re.findall('共(.*?)页',str(data))        for i in range(0, int(pages[0])):            # 构造博客专家所有博文翻页url            purl = str(response.url) + '/article/list/' + str(i+1)            yield Request(url= purl, callback=self.article)    def article(self, response):        # 爬取博主当前页所有文章url        articles = response.xpath('//span[@class="link_title"]/a/@href').extract()        for aurl in articles:            url = "http://blog.csdn.net" + aurl            yield Request(url=url, callback=self.detail)    def detail(self, response):        item = CsdnblogItem()        # 爬取博文详情页信息        item['url'] = response.url        # 新版主题CSDN和旧版主题CSDN按照不同抓取规则抓取        title = response.xpath('//span[@class="link_title"]/a/text()').extract()        if not title:            item['title'] = response.xpath('//h1[@class="csdn_top"]/text()').extract()[0].encode('utf-8')            item['releaseTime'] = response.xpath('//span[@class="time"]/text()').extract()[0].encode('utf-8')            item['readnum'] = response.xpath('//button[@class="btn-noborder"]/span/text()').extract()[0]        else:            item['title'] = title[0].encode('utf-8').strip()            item['releaseTime'] = response.xpath('//span[@class="link_postdate"]/text()').extract()[0]            item['readnum'] = response.xpath('//span[@class="link_view"]/text()').extract()[0].encode('utf-8')[:-9]        # 抓取正文        data = response.xpath('//div[@class="markdown_views"]|//div[@id="article_content"]')        item['article'] = data.xpath('string(.)').extract()[0]        # 用python jieba模块提取正文关键字，topK=2表示提取词频最高的两个        tags = jieba.analyse.extract_tags(item['article'], topK=2)        item['keywords'] = (','.join(tags))        print "博文标题: ", item['title']        print "博文链接: ", item['url']        yield item

5. 运行项目:

scrapy crawl spider_csdnblog --nolog

6. 在 MongoDB 客户端中查看数据:

Note: 如果要爬取指定博主的所有博文，只需要简单修改爬虫文件即可(保留 total 函数及其下面函数)。

至此，CSDN 博客专家的所有博文已成功爬取并存入了 MongoDB 数据库中。