Python爬虫学习(4)Scrapy采集框架

Scrapy入门
<strong>注意: Python版本需要为2.7</strong>

<strong>丛书编者按</strong>:Scrapy由 Python 编写。如果刚接触并且好奇这门语言的特性以及Scrapy的详情，对于已经熟悉其他语言并且想快速学习Python的编程老手， Learn Python The Hard Way ，对于想从Python开始学习的编程新手，非程序员的Python学习资料列表将是您的选择。

1.定义Item爬取模型

首先根据需要从dmoz.org获取到的数据对item进行建模。我们需要从dmoz中获取名字，url，以及网站的描述。对此，在item中定义相应的字段。编辑 tutorial 目录中的 items.py 文件:

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

# Creator yuluoxinsheng
import scrapy


class WikiItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field()

2.编写第一个爬虫(Spider)

为了创建一个Spider，必须继承 scrapy.Spider
类，且定义以下三个属性:
name
: 用于区别Spider。该名字必须是唯一的，您不可以为不同的Spider设定相同的名字
allowed_domains :
代表允许执行的url范围，通常以http请求字段的orion url为基准
start_urls
: 包含了Spider在启动时进行爬取的url列表。因此，第一个被获取到的页面将是其中之一。后续的URL则从初始的URL获取到的数据中提取。
parse()
是spider的一个方法。被调用时，每个初始URL完成下载后生成的 Response
对象将会作为唯一的参数传递给该函数。该方法负责解析返回的数据(response data)，提取数据(生成item)以及生成需要进一步处理的URL的 Request
对象。

import scrapy

class WikiSpider(scrapy.Spider):
    name = "wiki"
    allowed_domains = ["domz.org"]
    start_urls = [
        "http://www.dmoztools.net/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoztools.net/Computers/Programming/Languages/Python/Resources/"
    ]

     def parse(self, response):
         filename = response.url.split("/")[-2]
         with open(filename,'wb') as f:
             f.write(response.body)

3.创建wikipedia scrapy

> scrapy startproject wiki

console output

error.png

又遇到了windows的环境变量问题，题主也是无奈,尝试解决方法

成功解决：

solve.png

爬取 (crawl)

<h1> Using the scrapy tool </h1>
Scrapy X.Y - no active project
Usage:
scrapy <command> [options] [args]
Available commands:
crawl Run a spider
fetch Fetch a URL using the Scrapy downloader
[...]

进入项目的根目录(题主的目录为e:\Spider\wiki)，执行下列命令启动spider:

> scrapy crawl wiki

console output

SpiderBoot.png

request.png

查看包含 [wiki] 的输出，输出的log中包含定义在 start_urls 的初始URL，并且与spider中是一一对应的。可以看到发出的get请求指向我们创建的两条链接,除此之外没有指向其他页面

<strong>4.Spider解析数据:</strong>
1.Scrapy为Spider的 start_urls属性中的每个URL创建了 scrapy.Request
对象，并将 parse 方法作为回调函数 (callback)赋值给了Request。
2.Request对象经过调度，执行生成 scrapy.http.Response
对象并送回给spider parse()
方法。

进入项目的根目录，执行下列命令来启动shell用来加载解析的页面数据:

> scrapy shell "http://www.dmoztools.net/Computers/Programming/Languages/Python/Books/"

当shell载入后，将得到一个包含response数据的本地 response 变量。我们之前在parse函数中写入的response,body。shell显示的就是从domz爬取得body信息

console output

response.png

但这样解析出的数据某些不符合我们的需求，比如我们只要求回调/Languages的资源地址。而当输入 response.selector 时，获取到一个可以用于查询返回数据的selector(选择器)，以及映射到 response.selector.xpath() 、 response.selector.css() 的快捷方法(shortcut): response.xpath() 和 response.css() 。

selector根据response的类型自动选择最合适的分析规则(XML vs HTML)。

5.以Domz为例，老样子解析页面结构

domz.png

标签层次调用如下:
title aside->div->h2->a->font
link aside->div->h2->a href
desc aside->div->h3->a->font

更改Spider\WikiSpider，由于Domz网站域名更新，修改url

import scrapy

from wikiSpider.items import WikispiderItem



class WikiSpider(scrapy.Spider):
    name = "wiki"
    allowed_domains = ["dmoztools.net"]
    start_urls = [
        "http://dmoztools.net/Computers/Programming/Languages/Python/Books/",
        "http://dmoztools.net/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        item = WikispiderItem()
        content = response.xpath('//section/aside/div/text()')
        print content
        for sel in response.xpath('//section/aside/div'):
            item['title'] = sel.xpath('h2/a/text()')
            item['link'] = sel.xpath('h2/a/@href')
            item['desc'] = sel.xpath('h3/a/text()')
            return item

命令重启动spider:

> scrapy crawl wiki

console output :Error:xpath无法解析路径

error_first.jpg

Python console
启动shell，解析Books页面

scrapy shell http://dmoztools.net/Computers/Programming/Languages/Python/Books/

6.测试xpath获取页面各个标签值

challage.jpg

Test Result:

发现解析页面URL为 DomzBook
response.xpath('//section/aside/div')未获取到响应标签内容，尝试xpath语法

xpath.jpg

7.重新解析页面层次调用结构

"title" div class="title-and-desc" -> a -> div class="site-title" -> text()

"link" div class="title-and-desc" -> a -> @href
"desc" div class="site-descr" -> text()

console Input

response.xpath("//div[@class='title-and-desc']")

更新WikiSpider.py

import scrapy

from wikiSpider.items import WikispiderItem



class WikiSpider(scrapy.Spider):
    name = "wiki"
    allowed_domains = ["dmoztools.net"]
    start_urls = [
        "http://dmoztools.net/Computers/Programming/Languages/Python/Books/",
        "http://dmoztools.net/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        item = WikispiderItem()
        content = response.xpath("//div[contains(@class,'site-item')]/text()")
        print content
        for sel in response.xpath("//div[contains(@class,'site-item')]"):
            item['title'] = sel.xpath("div[@class='title-and-desc']/a/div[@class='site-title']/text()")
            item['link'] = sel.xpath("div[@class='title-and-desc']/a/@href")
            item['desc'] = sel.xpath("div[@class='site-descr']/text()")
            return item

console output

response.jpg

发现某些div未正确解析，查看嵌套选择器,原因在于Xpath是一种基于XML文档的搜索方式，对于css的规则仅解析为字符串，尝试contains标记属性节点

console Input
response.xpath("//div[contains(@class,'site-item')]")

console output > > 标注行已成功返回响应标签值

F:\pythonProject\wikiSpider>scrapy crawl wiki
2017-07-09 22:24:14 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: wikiSpider)
2017-07-09 22:24:14 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'wikiSpider.spiders', 'SPIDER_MODULES': ['wikiSpider.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'wikiSpider'}
2017-07-09 22:24:14 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2017-07-09 22:24:15 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-07-09 22:24:15 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-07-09 22:24:15 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-07-09 22:24:15 [scrapy.core.engine] INFO: Spider opened
2017-07-09 22:24:15 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-07-09 22:24:15 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6027
2017-07-09 22:24:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://dmoztools.net/robots.txt> (referer: None)
2017-07-09 22:24:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://dmoztools.net/Computers/Programming/Languages/Python/Books/> (referer: None)
[<Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
'>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
   '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
      '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
         '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
               '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
                  '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
                     '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
                        '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
                           '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
                              '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u
'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'
\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()"
data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" d
ata=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/te
xt()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/tex
t()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item
')]/text()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item'
)]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'sit
e-item')]/text()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site
-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@clas
s,'site-item')]/text()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class
,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains
(@class,'site-item')]/text()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(
@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[co
ntains(@class,'site-item')]/text()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[con
tains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//
div[contains(@class,'site-item')]/text()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//d
iv[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpa
th="//div[contains(@class,'site-item')]/text()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpat
h="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Select
or xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selecto
r xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <
Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <S
elector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n
 '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                            '>]
2017-07-09 22:24:17 [scrapy.core.scraper] DEBUG: Scraped from <200 http://dmoztools.net/Computers/Programming/Languages/Python/Books/>
{'desc': [],
 'link': [<Selector xpath="div[@class='title-and-desc']/a/@href" data=u'http://www.brpreiss.com/books/opus7/html'>],
 'title': [<Selector xpath="div[@class='title-and-desc']/a/div[@class='site-title']/text()" data=u'Data Structures and Algorithms with Obje'>]}
2017-07-09 22:24:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://dmoztools.net/Computers/Programming/Languages/Python/Resources/> (referer: None)
[<Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
'>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
   '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
      '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
         '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
            '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
               '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
                  '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n\r\n                                '>, <Selector xpath="//div[contains(@class,'site-item')]/text()" data=u'\r\n
                     '>]
> > 2017-07-09 22:24:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://dmoztools.net/Computers/Programming/Languages/Python/Resources/>
{'desc': [],
 'link': [<Selector xpath="div[@class='title-and-desc']/a/@href" data=u'http://www.pythonware.com/daily/'>],
 'title': [<Selector xpath="div[@class='title-and-desc']/a/div[@class='site-title']/text()" data=u"eff-bot's Daily Python URL ">]}
2017-07-09 22:24:19 [scrapy.core.engine] INFO: Closing spider (finished)

最后编辑于：2017.12.08 04:54:46

人面猴
序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
沈念sama阅读 204,053评论 6赞 478
死咒
序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
沈念sama阅读 85,527评论 2赞 381
救了他两次的神仙让他今天三更去死
文/潘晓璐我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
开封第一讲书人阅读 150,779评论 0赞 337
道士缉凶录：失踪的卖姜人
文/不坏的土叔我叫张陵，是天一观的道长。经常有香客问我，道长，这世上最难降的妖魔是什么？我笑而不...
开封第一讲书人阅读 54,685评论 1赞 276
港岛之恋（遗憾婚礼）
正文为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
茶点故事阅读 63,699评论 5赞 366
恶毒庶女顶嫁案：这布局不是一般人想出来的
文/花漫我一把揭开白布。她就那样静静地躺着，像睡着了一般。火红的嫁衣衬着肌肤如雪。梳的纹丝不乱的头发上，一...
开封第一讲书人阅读 48,609评论 1赞 281
城市分裂传说
那天，我揣着相机与录音，去河边找鬼。笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。我是一名探鬼主播，决...
沈念sama阅读 37,989评论 3赞 396
双鸳鸯连环套：你想象不到人心有多黑
文/苍兰香墨我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
开封第一讲书人阅读 36,654评论 0赞 258
万荣杀人案实录
序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
沈念sama阅读 40,890评论 1赞 298
护林员之死
正文独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋以下内容为张勋视角年9月15日...
茶点故事阅读 35,634评论 2赞 321
白月光启示录
正文我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
茶点故事阅读 37,716评论 1赞 330
活死人
序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
沈念sama阅读 33,394评论 4赞 319
日本核电站爆炸内幕
正文年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
茶点故事阅读 38,976评论 3赞 307
男人毒药：我在死后第九天来索命
文/蒙蒙一、第九天我趴在偏房一处隐蔽的房顶上张望。院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
开封第一讲书人阅读 29,950评论 0赞 19
一桩弑父案，背后竟有这般阴谋
文/苍兰香墨我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。一阵脚步声响...
开封第一讲书人阅读 31,191评论 1赞 260
情欲美人皮
我被黑心中介骗来泰国打工，没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。一个月前我还...
沈念sama阅读 44,849评论 2赞 349
代替公主和亲
正文我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
茶点故事阅读 42,458评论 2赞 342

Python爬虫学习(4)Scrapy采集框架

推荐阅读更多精彩内容