scrapy版本：1.5.1
scrapy-redis版本：0.6.8
redis版本：2.10.6
scrapy_redis的git：https://github.com/rmax/scrapy-redis
该项目的git：https://github.com/MsLpoi/sr_demo
在该系列第一篇的基础上，我们继续来编写从爬虫吧~
1. 开始
基于上一篇《【scrapy_redis】简单分布式爬虫1》的sr_demo项目
1. setting.py
1. 在文件的最后添加以下代码（就是scrapy_redis的git首页里面的设置代码，但把它的ITEM_PIPELINES的设置删掉了）
# Enables scheduling storing requests queue in redis.
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
# Ensure all spiders share same duplicates filter through redis.
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
# Default requests serializer is pickle, but it can be changed to any module
# with loads and dumps functions. Note that pickle is not compatible between
# python versions.
# Caveat: In python 3.x, the serializer must return strings keys and support
# bytes as values. Because of this reason the json or msgpack module will not
# work by default. In python 2.x there is no such issue and you can use
# 'json' or 'msgpack' as serializers.
#SCHEDULER_SERIALIZER = "scrapy_redis.picklecompat"
# Don't cleanup redis queues, allows to pause/resume crawls.
#SCHEDULER_PERSIST = True
# Schedule requests using a priority queue. (default)
#SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.PriorityQueue'
# Alternative queues.
#SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.FifoQueue'
#SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.LifoQueue'
# Max idle time to prevent the spider from being closed when distributed crawling.
# This only works if queue class is SpiderQueue or SpiderStack,
# and may also block the same time when your spider start at the first time (because the queue is empty).
#SCHEDULER_IDLE_BEFORE_CLOSE = 10
# The item pipeline serializes and stores the items in this redis key.
#REDIS_ITEMS_KEY = '%(spider)s:items'
# The items serializer is by default ScrapyJSONEncoder. You can use any
# importable path to a callable object.
#REDIS_ITEMS_SERIALIZER = 'json.dumps'
# Specify the host and port to use when connecting to Redis (optional).
#REDIS_HOST = 'localhost'
#REDIS_PORT = 6379
# Specify the full Redis URL for connecting (optional).
# If set, this takes precedence over the REDIS_HOST and REDIS_PORT settings.
REDIS_URL = 'redis://localhost:6379'
# Custom redis client parameters (i.e.: socket timeout, etc.)
#REDIS_PARAMS = {}
# Use custom redis client class.
#REDIS_PARAMS['redis_cls'] = 'myproject.RedisClient'
# If True, it uses redis' ``SPOP`` operation. You have to use the ``SADD``
# command to add URLs to the redis queue. This could be useful if you
# want to avoid duplicates in your start urls list and the order of
# processing does not matter.
#REDIS_START_URLS_AS_SET = False
# Default start urls key for RedisSpider and RedisCrawlSpider.
#REDIS_START_URLS_KEY = '%(name)s:start_urls'
# Use other encoding than utf-8 for redis.
#REDIS_ENCODING = 'latin1'
里面的设置项很多，但我们目前只需关注REDIS_HOST、REDIS_PORT、REDIS_URL。
1）REDIS_HOST、REDIS_PORT：这两个是搭配使用的，分别是设置需要连接的redis数据库的HOST和PORT。
2）REDIS_URL：这也是用于需要设置连接的redis数据库的，但是它能设置的除了HOST和PORT，还有很多别的设置项，建议大家自行了解并按照需要修改。
注：第1项和第二项不会同时生效，优先级：(REDIS_URL) > (REDIS_HOST、REDIS_PORT)，建议直接设置REDIS_URL。
2. read_urls.py
在sr_demo项目下使用以下命令创建一个新的爬虫
scrapy genspider -t basic read_urls news.stcn.com
并在该文件输入以下代码
# -*- coding: utf-8 -*-
from scrapy_redis.spiders import RedisSpider
class ReadUrlsSpider(RedisSpider):
name = 'read_urls'
redis_key = 'read_urls:start_urls'
custom_settings = {
'ITEM_PIPELINES': {
'scrapy_redis.pipelines.RedisPipeline': 300,
}
}
def parse(self, response):
url = response.url
title = response.xpath('//div[@class="intal_tit"]/h2/text()').exctart_frist()
tmp = {'url': url, 'title': title}
return tmp
- 这里继承的是RedisSpider类（当然还有RedisCrawlSpider类，我没用而已）。
- redis_key：爬虫将会在redis数据库中的redis_key列表中获取待爬取的链接，因此我们直接输入之前得到的read_urls:start_urls列表就好了。
3. 运行
运行read_urls爬虫，并查看数据库，没错的话就成功啦~
scrapy crawl read_urls
思考：
1. 将setting.py中的SCHEDULER_PERSIST设置为False，再运行get_urls、read_urls爬虫，分别观察redis数据库内容变化。
2. 当爬取完链接之后，爬虫会一直空跑（实际上是一直在等待read_urls:start_urls列表中的列表）。这时候再运行一遍read_urls爬虫，再观察输出。
