4、scrapy内置服务

来源：互联网发布：手机淘宝不能追加评论编辑：程序博客网时间：2024/06/15 17:12

本篇介绍scrapy的Logging模块、Stats Collections模块和Sending-email模块。

Logging

简单使用方法

import logginglogging.warning("This is a Warning")

执行结果为：

WARNING:root:This is a Warning

通用的记录日志的方法，可加入日志的级别

import logginglogging.log(logging.WARNING, "This is a Warning")

通用logger记录日志

import logginglogger = logging.getLogger(__name__)logger.warning("This is a Warning")

运行结果:

WARNING:main:This is a warning

在scrapy中使用logging

import scrapyclass MySpider(scrapy.Spider):    name = 'myspider'    start_urls = ['http://scrapyinghub.com']    def parse(self, response):        self.logger.info('Parse function called on %s' % response.url)

settings中设置log

LOG_FILE
LOG_ENABLE
LOG_ENCODING
LOG_LEVEL
LOG_FORMAT
lOG_DATEFORMAT
lOG_STDOUT

Stats Collections

使用方法（待学习）

class ExtensionThatAccessStats(object):    def __init__(self, stats):        self.stats = stats    @classmethod    def from_crawler(cls, crawler):        return cls(crawler.stats)

scrapy内置的stats Collections

class scrapy.statscollectors.MemoryStatsCollector(linux)
class scrapy.statscollectors.DummyStatsCollector

Sending-email

通用使用方法

from scrapy.mail import MailSendermailer = MailSender()# mailer = MailSender.from_settings(settins)mailer.send(to=["someone@example.com"], subject="Some subject", body="Some body", cc=["another.example.com"])

0 0