技术栈
Scrapy（Stats Collection）
业务背景
最近在用Scrapy爬取某平台的小区信息时，为保证小区数据的完整性，同时减少工作量，避免重复造轮子，所以决定通过Scrapy自带的Scrapy Stats Collection 验证数据的完整性，同时监控爬虫应用的性能。
Stats Collection
官方文档介绍
Scrapy provides a convenient facility for collecting stats in the form of key/values, where values are often counters. The facility is called the Stats Collector, and can be accessed through the stats attribute of the Crawler API
The Stats Collector keeps a stats table per open spider which is automatically opened when the spider is opened, and closed when the spider is closed.
Scrapy提供了一种方便的工具（stats collector），可以以键/值的形式收集统计信息，其中值通常是计数器, 可以通过Crawler API 访问
默认状态收集器
MemoryStatsCollector
继承自基本的状态收集器StatsCollector
A simple stats collector that keeps the stats of the last scraping run (for each spider) in memory, after they’re closed. The stats can be accessed through the spider_stats attribute, which is a dict keyed by spider domain name.
This is the default Stats Collector used in Scrapy.
spider_stats
A dict of dicts (keyed by spider name) containing the stats of the last scraping run for each spider.
在基础功能的基础上，增加了spider_stats属性，保存上次运行的每个爬虫的状态，保存在内存中。
Stats数据示例
{'downloader/request_bytes': 444,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 2701,
'downloader/response_count': 2,
'downloader/response_status_count/200': 1,
'downloader/response_status_count/404': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 9, 6, 12, 24, 28, 276209),
'log_count/DEBUG': 3,
'log_count/INFO': 7,
'response_received_count': 2,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2018, 9, 6, 12, 24, 26, 236622)}
RetryMiddleware
A middleware to retry failed requests that are potentially caused by temporary problems such as a connection timeout or HTTP 500 error.
retry 配置与使用
1. RETRY_ENABLED
2. RETRY_TIMES
3. RETRY_HTTP_CODES
4. RETRY_PRIORITY_ADJUST
5. spider中重试 get_retry_request()
The RetryMiddleware can be configured through the following settings (see the settings documentation for more info)
RETRY_ENABLED
Default: True
Whether the Retry middleware will be enabled.
RETRY_TIMES
Default: 2
Maximum number of times to retry, in addition to the first download.
Maximum number of retries can also be specified per-request using max_retry_times attribute of Request.meta. When initialized, the max_retry_times meta key takes higher precedence over the RETRY_TIMES setting.
RETRY_HTTP_CODES
Default: [500, 502, 503, 504, 522, 524, 408, 429]
Which HTTP response codes to retry. Other errors (DNS lookup issues, connections lost, etc) are always retried.
In some cases you may want to add 400 to RETRY_HTTP_CODES because it is a common code used to indicate server overload. It is not included by default because HTTP specs say so.
RETRY_PRIORITY_ADJUST
Default: -1
Adjust retry request priority relative to original request:
a positive priority adjust means higher priority.
a negative priority adjust (default) means lower priority.
If Request.meta has dont_retry key set to True, the request will be ignored by this middleware.
To retry requests from a spider callback, you can use the get_retry_request() function
1. 请求重试时会被记录到stats collector
2.Scrapy 默认启用Retry middleware 通过配置RETRY_ENABLED为True.
Question1: RetryMiddleware的执行顺序？
RetryMiddleware属于Downloader Middlewares
Each downloader middleware is a Python class that defines one or more of the methods defined below.(自定制中间件可以定义以下一个或多个方法)
process_request(request, spider)：process_request() should either: return None, return a Response object, return a Request object, or raise IgnoreRequest.
process_response(request, response, spider)： process_response() should either: return a Response object, return a Request object or raise a IgnoreRequest exception.
process_exception(request, exception, spider)：Scrapy calls process_exception() when a download handler or a process_request() (from a downloader middleware) raises an exception (including an IgnoreRequest exception)，process_exception() should return: either None, a Response object, or a Request object. （当下载器和下载器中间件的process_request抛出异常时会调用process_excetion）
Scrapy基本的中间件优先级
DOWNLOADER_MIDDLEWARES_BASE
{
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100,
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300,
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350,
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500,
'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,
'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560,
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590,
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600,
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
'scrapy.downloadermiddlewares.stats.DownloaderStats': 850,
'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
}
A dict containing the downloader middlewares enabled by default in Scrapy. Low orders are closer to the engine, high orders are closer to the downloader. You should never modify this setting in your project, modify DOWNLOADER_MIDDLEWARES instead. For more info see Activating a downloader middleware.
可以看到重试中间件的优先级是550，值越高距离下载器越近，因此多个DOWNLOADER_MIDDLEWARES执行顺序如下
Answer1
process_request 执行顺序: DOWNLOADER_MIDDLEWARE 值越小，越优先执行；
process_response 执行顺序: DOWNLOADER_MIDDLEWARE 值越大，越优先执行；
也就是说当自定制的DOWNLOADER_MIDDLEWARES和RetryMiddleware（默认）同时激活时
Question2:
假设激活自定制中间件A（作用是对500 Response的特殊处理）的优先级是200，默认重试中间件B的优先级是550( RETRY_HTTP_CODES 重试)，当response状态码是500，是重试还是执行特殊逻辑呢？
Answer2:
process_response 优先级大的优先执行，因此会先执行中间件B的重试逻辑，再执行自定制中间件A的特殊逻辑
