Scrapy Redis源码 spider分析

下载的scrapy-redis源码中的spiders.py源码非常分析：

RedisSpider继承了Spider和RedisMixin这两个类，RedisMixin是用来从redis读取url的类。
当我们生成一个Spider继承RedisSpider时，调用setup_redis函数，这个函数会去连接redis数据库，然后会设置signals，一个是当spider空闲的时候(signal)，会调用spider_idle函数，这个函数调用schedule_next_request函数，然后抛出DontCloseSpider异常，保证spider是一直活着的状态；另一个signal是当抓到一个item时，item_scraped函数，这个函数会调用schedule_next_request函数。

from scrapy import Spider, signalsfrom scrapy.exceptions import DontCloseSpiderfrom . import connectionclass RedisMixin(object):    """Mixin class to implement reading urls from a redis queue."""    redis_key = None  # use default '<spider>:start_urls'    def setup_redis(self):        """Setup redis connection and idle signal.        This should be called after the spider has set its crawler object.        """        if not self.redis_key:            self.redis_key = '%s:start_urls' % self.name        self.server = connection.from_settings(self.crawler.settings)        # idle signal is called when the spider has no requests left,        # that's when we will schedule new requests from redis queue        self.crawler.signals.connect(self.spider_idle, signal=signals.spider_idle)        self.crawler.signals.connect(self.item_scraped, signal=signals.item_scraped)        self.log("Reading URLs from redis list '%s'" % self.redis_key)    def next_request(self):        """Returns a request to be scheduled or none."""        use_set = self.settings.getbool('REDIS_SET')        if use_set:            url = self.server.spop(self.redis_key)        else:            url = self.server.lpop(self.redis_key)        if url:            return self.make_requests_from_url(url)    def schedule_next_request(self):        """Schedules a request if available"""        req = self.next_request()        if req:            self.crawler.engine.crawl(req, spider=self)    def spider_idle(self):        """Schedules a request if available, otherwise waits."""        self.schedule_next_request()        raise DontCloseSpider    def item_scraped(self, *args, **kwargs):        """Avoids waiting for the spider to  idle before scheduling the next request"""        self.schedule_next_request()class RedisSpider(RedisMixin, Spider):    """Spider that reads urls from redis queue when idle."""    def _set_crawler(self, crawler):        super(RedisSpider, self)._set_crawler(crawler)        self.setup_redis()

我们下到的scrapy-redis源码中有自带一个example-project项目，这个项目包含3个spider，分别是dmoz, mycrawler_redis, myspider_redis，

dmoz（class DmozSpider(CrawlSpider)）这个爬虫继承的是CrawlSpider，它是用来说明Redis的持续性，当我们第一次运行dmoz爬虫，然后Ctrl-C停掉之后，再运行dmoz爬虫，之前的爬取记录是保留下来的。

myspider_redis ( class MySpider(RedisSpider) ) 这个爬虫继承了RedisSpider， 它能够支持分布式的抓取，采用的是basic spider，需要parse函数

mycrawler_redis ( class MyCrawler(RedisMixin, CrawlSpider) ) 这个爬虫，继承了RedisMixin和CrawlSpider，能够支持分布式的抓取，采用的是crawlSpider，需要设置Rule规则