python网络爬虫笔记三

一、Scrapy爬虫框架常用命令

命令 说明 格式
startproject 创建一个新工程 scrapy startproject <name> [dir]
genspider 创建一个爬虫 scrapy genspider [options]<name> <domain>
settings 获取爬虫配置信息 scrapy settings [options]
crawl 运行一个爬虫 scrapy crawl <spider>
list 列出工程中所有爬虫 scrapy list
shell 启动URL调试命令行 scrapy shell [url]

二、scrapy使用

  1. scrapy startproject demo 新建一个项目demo
  2. scrapy genspider spiderdemo www.baidu.com 创建一个百度的爬虫
  3. scrapy crawl spiderdemo 运行爬虫

三、Request类

方法 说明
.url Request对应的URL
.method 'GET' 'POST'等
.headers 字典类型风格的请求头
.body 请求内容主体，字符串类型
.meta 用户添加的扩展信息，scrapy内部模块间传递
.copy() 复制请求

四、Response类

方法 说明
.url Response对应的URL
.status HTTP状态码
.headers Response的头部
.body Response内容主体，字符串类型
.flags 一组标记
.request 产生对应的Request对象
.copy() 复制响应

五、Scrapy框架的股票数据定向爬虫

# pipelines.py
# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html


class Python123DemoPipeline(object):
    def process_item(self, item, spider):
        return item

class BaidustocksPipeline(object):
    def open_spider(self, spider):
        self.f = open('Stock.txt', 'w')

    def close_spider(self,spider):
        self.f.close()

    def process_item(self, item, spider):
        try:
            line = str(dict(item)) + '\n'
            self.f.write(line)
        except:
            pass
        return item

# demo.py
# -*- coding: utf-8 -*-
import scrapy
import re

class DemoSpider(scrapy.Spider):
    name = 'demo'
    # allowed_domains = ['python123.io']
    start_urls = ['http://quote.eastmoney.com/stocklist.html']
    stock_info_url = 'https://gupiao.baidu.com/stock/'

    def parse(self, response):
        for href in response.css('a::attr(href)').extract():
            try:
                stock = re.findall(r"[s][hz]\d{6}", href)[0]
                url = 'https://gupiao.baidu.com/stock/' + stock + '.html'
                yield scrapy.Request(url, callback=self.parse_stock)
            except:
                continue

    def parse_stock(self, response):
        infoDict = {}
        stockInfo = response.css('.stock-bets')
        name = stockInfo.css('.bets-name').extract()[0]
        keyList = stockInfo.css('dt').extract()
        valueList = stockInfo.css('dd').extract()
        for i in range(len(keyList)):
            key = re.findall(r'>.*</dt>', keyList[i])[0][1:-5]
            try:
                val = re.findall(r'\d+\.?.*</dd>', valueList[i])[0][0:-5]
            except:
                val = "--"
        infoDict.update({'股票名称':re.findall('\s.*\(', name)[0].split()[0]
                        + re.findall('\>.*\<', name)[0][1:-1]}
                        )
        yield infoDict

# settings.py
# -*- coding: utf-8 -*-

# Scrapy settings for python123demo project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'python123demo'

SPIDER_MODULES = ['python123demo.spiders']
NEWSPIDER_MODULE = 'python123demo.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'python123demo (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'python123demo.middlewares.Python123DemoSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'python123demo.middlewares.Python123DemoDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'python123demo.pipelines.BaidustocksPipeline': 300,
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
【社区内容提示】社区部分内容疑似由AI辅助生成，浏览时请结合常识与多方信息审慎甄别。
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。

相关阅读更多精彩内容

  • Python网络爬虫
    Python开发简单爬虫（Python2.X版本，Eclipse工具） 一、爬虫介绍 爬虫调度端：启动、停止爬虫，...
    凛0_0阅读 2,211评论 2赞 10
  • 9.2 scrapy安装及基本使用
    https://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml 下载即可。 安装...
    怂恿的大脑阅读 1,414评论 0赞 7
  • scrapy学习笔记(有示例版）
    scrapy学习笔记(有示例版） 我的博客 scrapy学习笔记1.使用scrapy1.1创建工程1.2创建爬虫模...
    陈思煜阅读 13,182评论 4赞 46
  • Python网络爬虫
    1. 概述 本文主要介绍网络爬虫，采用的实现语言为Python，目的在于阐述网络爬虫的原理和实现，并且对目前常见的...
    Lemon_Home阅读 3,140评论 0赞 21
  • 无标题文章
    我发现自己不能很主动的去思考或者发现我在工作中遇到的问题，而老是抱着不懂就过的心态，这样很不好。
    here_30e0阅读 199评论 0赞 0

友情链接更多精彩内容