scrapy爬取糗事百科入门

1.setting配置
(1)在创建的setting文件中首先需要将ROBOTSTXT_OBEY参数设置为false,这样防止程序在查找不到机器人协议文件以后停止爬取
(2)在DEFAULT_REQUEST_HEADERS参数中将user-agent请求头加入进去
2.建议在items.py中定义好各个字段,使用items中的类不不使用dict
3.pipelines中共有三个函数分别是open_spider:爬虫开始时执行,process_item:数据处理时执行, close_spider爬虫结束时执行
当pipelines编写完成时需要在setting中ITEM_PIPELINES参数设置pipeline的等级
4.在spider中每生成一个items可以用yield将每个items传递给pipelines, 如果不使用yield可以将所有的items放到一个列表中,将列表return,之后会将列表中的元素一个一个返回给pipelines

5.笔记:

笔记.png

6.代码示例
(1)spiders

# -*- coding: utf-8 -*-
import scrapy
from scrapy_demo.items import ScrapyDemoItem

class QsbkSpider(scrapy.Spider):
    name = 'qsbk'
    allowed_domains = ['qiushibaike.com']
    start_urls = ['https://www.qiushibaike.com/text/page/1/']

    def parse(self, response):
        div_content = response.xpath('//div[@class="col1 old-style-col1"]/div')
        for div in div_content:
            author = div.xpath('.//div[@class="author clearfix"]//h2/text()').extract()
            content = div.xpath('.//div[@class="content"]/span//text()').extract()
            print('作者: {}'.format(author[0].strip()))
            print('内容: {}'.format(''.join(content).strip()))
            items = ScrapyDemoItem(author=author, content=content)
            yield items

pipelines.py

# -*- coding: utf-8 -*-
from json import dumps
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


class ScrapyDemoPipeline(object):
    def open_spider(self, spider):
        self.f = open('funs.json', 'w', encoding='utf-8')
        print('爬虫开始了...', spider)

    def process_item(self, item, spider):
        print('正在进行数据处理...', spider)
        self.f.write(dumps(dict(item), ensure_ascii=False) + '\n')
        return item

    def close_spider(self, spider):
        self.f.close()
        print('爬虫结束了...', spider)

items.py

import scrapy


class ScrapyDemoItem(scrapy.Item):
    author = scrapy.Field()
    content = scrapy.Field()