scrapy配置redis简单实例

安装

1. 安装库

pip install scrapy-redis

2. 安装redis

配置

在setting.py文件中配置

SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderQueue'
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_HOST = 'localhost'
REDIS_PORT = 6379

编写spider

class MySpider(RedisSpider):
    name = "myspider"
    start_urls = ["http://weather.sina.com.cn/", ]
    redis_key = "myspider:start_urls"
 
    # 注意一定要重写start_requests,否则出不来数据
    def start_requests(self):
        for url in self.start_urls:
            yield Request(url=url, callback=self.parse)
 
    def parse(self, response):
        .....

上述中一定要注意重写start_requests，不然一直没数据出来，这是本人走过的坑，希望能帮到您，如果你觉得这篇文章对你有帮助，麻烦点个赞哦！