一、准备工作
1.安装splash
在windows环境下，splash可通过docker进行安装，安装方法在 这篇文章 中有详细讲解，在此不再赘述。
安装完成后在powershell中运行以下命令安装splash：
#从docker hub下载相关镜像文件
docker pull scrapinghub/splash、
使用docker启动服务命令启动Splash服务，在8050、8051、5023端口开启splash服务：
#启动splash服务，并通过http，https，telnet提供服务
#通常一般使用http模式 ，可以只启动一个8050就好
#Splash 将运行在 0.0.0.0 at ports 8050 (http), 8051 (https) and 5023 (telnet).
docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash
在浏览器中输入10.0.75.1:8050查看服务启动情况：
QQ截图20180214144552.png
2.安装scrapy-splash
使用pip安装scrapy-splash：
pip install scrapy-splash
完成以上步骤之后就可以在scrapy中使用splash了
二、编写爬虫
动态页面中的部分内容是浏览器运行页面中的javascript脚本生成的，相较于普通静态页面，爬取难度更大。以网站 http://quotes.toscrape.com/js/为例，爬取动态生成的名人名言信息。
在项目文件settings.py中配置scrapy-splash：
#定义splash服务器地址
SPLASH_URL = 'http://10.0.75.1:8050/'
#开启splash的下载中间件
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
#设置去重
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
编码实现：
# -*- coding: utf-8 -*-
import scrapy
from scrapy_splash import SplashRequest
class QuoteSpider(scrapy.Spider):
name = 'quote'
start_urls = ['http://quotes.toscrape.com/js/']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, args={'images': 0})
def parse(self, response):
quotes = response.xpath('//div[@class="quote"]')
for quote in quotes:
text = quote.xpath('.//*[@class="text"]/text()').extract_first()
author = quote.xpath('.//*[@class="author"]/text()').extract_first()
yield {
'text': text,
'author': author,
}
next_page = response.xpath('//a[contains(text(),"Next")]/@href').extract_first()
if next_page:
absolute_url = response.urljoin(next_page)
yield SplashRequest(absolute_url, args={'images': 0})
执行结果：
2018-02-14 15:17:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/>
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein'}
2018-02-14 15:17:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/>
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling'}
....
2018-02-14 15:17:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/page/10/>
{'text': "“A person's a person, no matter how small.”", 'author': 'Dr. Seuss'}
2018-02-14 15:17:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/page/10/>
{'text': '“... a mind needs books as a sword needs a whetstone, if it is to keep its edge.”', 'author': 'George R.R. Martin'}
结果显示我们已成功爬取动态生成的名人名言信息。
