Scrapy: 发送带Cookie的请求

Scrapy的Request类支持设置cookie属性，要在爬虫请求中带上cookie，可以重载Spider的start_requests方法。

import sysfrom scrapy.spider import Spiderfrom scrapy.selector import Selectorfrom scrapy.http.request import Requestclass InfoqSpider(Spider):    name = "techbrood"    allowed_domains = ["techbrood.com"]    start_urls = [        "http://techbrood.com",    ]            def start_requests(self):        for url in self.start_urls:                    yield Request(url, cookies={'techbrood.com': 'true'})

参考文档：

http://doc.scrapy.org/en/latest/topics/spiders.html?highlight=start_requests#scrapy.spider.Spider.start_requests

by iefreer