来源:天涯明月笙的慕课笔记

准备工作

系统windows7
安装MYSQL
提示:
- 安装的时候, 选安装选项server only
- 根据提示, 遇到安装界面没有下一步可以用键盘操作
  键盘操作
  b-back。n-next。x-execute。f-finish。c-cancel
- 根据界面完成安装, 进入安装目录下, mysqld -initialize命令初始化, 用'mysql -uroot -p'进入shell
- 用 net start mysql启动mysql服务, 如果服务名无效
  cmd打开到mysql/bin目录,输入 mysqld -install. 同时在控制面板进入服务选项, 启动mysql 服务. 多试试吧
安装pycharm
开启pycharm会员模式

伯乐在线爬取所有文章

安装模块

scrapy, pymysql, pillow, pypiwin32

pymysql是插入数据库的模块
用scrapy自带的ImagesPipeline需要pillow模块
创建爬虫后, windows输入命令scrapy crawl jobbole会报错需要pypiwin32

爬虫结构

items: 爬虫的解析信息的字段
包含名称, 设置输入输出处理器
pipelines: 爬虫的管道, 用于将解析后消息持久化存储
包含图片存储, Json文件的存储, 数据库的存储
settings: 爬虫各种相关设置
包含是否遵循ROBOTS_TXT, 爬虫下载网页时延, 爬虫图片下载存储的目录, 日志文件的存储目录, 管道的启用和优先级
spiders: 爬虫主体
爬虫的爬取主要逻辑

基本命令

# 创建爬虫项目
scrapy startproject jobbole_article 
# 进入spiders目录下, 生成爬虫
scrapy genspider jobbole blog.jobbole.com
# 运行爬虫
scrapy crawl jobbole

最终的文件目录, 上述命令后images文件夹暂时没有

伯乐在线爬虫目录.png

jobbole.py

# -*- coding: utf-8 -*-
import scrapy
from urllib import parse
from jobbole_article.items import ArticleItemLoader, JobboleArticleItem
from scrapy.http import Request


class JobboleSpider(scrapy.Spider):
    name = 'jobbole'
    allowed_domains = ['blog.jobbole.com']
    start_urls = ['http://blog.jobbole.com/all-posts']

    @staticmethod
    def add_num(value):
        return value if value else [0]

    def parse_deatail(self, response):
        response_url = response.url
        front_image_url = response.meta.get('front_image_url', '')
        item_loader = ArticleItemLoader(item=JobboleArticleItem(), response=response)
        item_loader.add_xpath('title', "//div[@class='entry-header']/h1/text()")
        item_loader.add_value('url', response_url)
        item_loader.add_value('url_object_id', response_url)
        item_loader.add_value('front_image_url', front_image_url)
        item_loader.add_xpath('content', "//div[@class='entry']//text()")
        # span_loader = loader.nested_path('//span[@class='href-style'])
        # 赞
        item_loader.add_xpath('praise_nums', "//span[contains(@class,'vote-post-up')]/h10/text()", self.add_num)
        # 评论
        item_loader.add_xpath('comment_nums', "//span[contains(@class, 'hide-on-480')]/text()", self.add_num)
        # 收藏
        item_loader.add_xpath('fav_nums', "//span[contains(@class, 'bookmark-btn')]/text()", self.add_num)
        item_loader.add_xpath('tags', "//p[@class='entry-meta-hide-on-mobile']/a[not(@href='#article-comment')]/text()")
        return item_loader.load_item()

    def parse(self, response):
        post_nodes = response.xpath("//div[@class='post floated-thumb']")
        for post_node in post_nodes:
            post_url = post_node.xpath(".//a[@title]/@href").extract_first("")
            img_url = post_node.xpath(".//img/@src").extract_first("")
            yield Request(url=parse.urljoin(response.url, post_url), meta={'front_image_url': img_url},
                          callback=self.parse_deatail)
        next_url = response.xpath('//a[@class="next page-numbers"]/@href').extract_first('')
        if next_url:
            yield Request(url=parse.urljoin(response.url, next_url), callback=self.parse)

模块

from urllib import parse
该模块主要用于对不完整的url进行补全

url = parse.urljoin('http://blog.jobbole.com/', '10000')
#url输出为拼接后的'http://blog.jobbole.com/10000', 如果后面是完整的, 则不拼接

from jobbole_article.items import ArticleItemLoader, JobboleArticleItem
是items.py中的类
from scrapy.http impot Request
- 构造scrapy网页请求, 请求需要跟进的url.
- meta参数,为字典形式. 主要是在Request中传送额外的变量给response.可以通过response.meta.get()获取
- callback参数则是请求内容下载完毕后调用相应的解析函数

比如在http://blog.jobbole.com/all-posts/中需要获取文章内容, 则构造对下面图片中箭头所指url的请求.内容下载完毕后调用parse_detail方法进行处理. 处理函数可以获得Request中键front_image_url的值img_url
对应代码

yield Request(url=parse.urljoin(response.url, post_url), meta={'front_image_url': img_url},
                          callback=self.parse_deatail)

Request.png

JobboleSpider类

该类继承scrapy.Spider, 其他的属性需要查看文档

@staticmethod
def add_num(value):

可暂时忽略,
该类的静态方法, 用在以下代码中, 作为输入处理器.主要作用是在解析相关字段为空值时返回默认值

item_loader.add_xpath('comment_nums', "//span[contains(@class, 'hide-on-480')]/text()", self.add_num)

自定义方法parse_detail

作用:解析文章详情页的,提取相关字段值的方法, 文章详情页如http://blog.jobbole.com/114420/. 返回填充后的item
一些变量的解释
response_url是响应内容的连接, 比如http://blog.jobbole.com/114420/
front_image_url是http://blog.jobbole.com/all-posts图片连接
item_loader是具有填充item方法的实例, 常用方法add_xpath, add_value, 注意填充后的item的值比如item['title']是一个列表
- add_xpath
  用xpath解析response的方法, 第一个参数如'title'是item的键或者说字段, 第二个是xpath解析规则, 第三个是处理器
- add_value
  直接赋予相应的值
- load_item
  执行填充item

JobboleSpider的自带parse方法

作用: 与parse_detail相同都是解析response, 不同的是parse是爬虫默认调用的解析方法.
response.xpath
xpath解析规则, 返回Selector对象,用extract()获取所有的文本值列表[], 或者是用extract_first()获取第一个文本值

xpath规则

一些规则
- 可以像url那样拼接规则, 但是注意的是第二个规则加.

post_nodes = response.xpath("//div[@class='post floated-thumb']")
        for post_node in post_nodes:
            #  .//a[@title]/@href
            post_url = post_node.xpath(".//a[@title]/@href").extract_first("")
            img_url = post_node.xpath(".//img/@src").extract_first("")

- xpath中不含某个属性"//div[not(@class='xx')]"
- xpath中包含某个属性"//div[contains(@class, 'xx')]" 
- @herf表示提取属性href的值, text()表示提取元素里的文本值
- //表示元素任意层下的子元素, /表示元素的直接子元素

调试方法
可以在浏览器中输入相应的路径测试, 但是要写css规则

css规则浏览器.png

用scrapy shell命令测试

scrapy shell http://blog.jobbole.com/all-posts
# 然后输入相应的规则可以看返回的值
response.xpath("...").extract()
# 可以用fetch(url)更改下载的Response
fetch('http://blog.jobbole.com/10000')

或者打断点,运行爬虫用pycharm可以查看

items.py

import scrapy
import re
import hashlib
from scrapy.loader import ItemLoader
from scrapy.loader.processors import TakeFirst, MapCompose, Join, Identity, Compose


def get_md5(value):
    if isinstance(value, str):
        value = value.encode(encoding='utf-8')
        # print('value--------------------------', value)
        m = hashlib.md5()
        m.update(value)
        return m.hexdigest()


def get_num(value):
    # print(value)
    if value:
        num = re.match(r".*?(\d+?)", value)
        try:
            # print("----------------",num.group(1), int(num.group(1)))
            return int(num.group(1))
        except (AttributeError, TypeError):
            return 0
    else:
        return 0

#多余
def return_num(value):
    # return value[0] if value else 0
    if value:
        return value
    else:
        return "1"


class JobboleArticleItem(scrapy.Item):
    # define the fields for your item here like:
    title = scrapy.Field()
    url = scrapy.Field()
    url_object_id = scrapy.Field(
        input_processor=MapCompose(get_md5)
    )
    front_image_url = scrapy.Field(
        output_processor=Identity()
    )
    front_image_path = scrapy.Field()
    content = scrapy.Field(
        output_processor=Join()
    )
    praise_nums = scrapy.Field(
        input_processor=MapCompose(get_num),
        # output_processor=MapCompose(return_num)
    )
    fav_nums = scrapy.Field(
        input_processor=MapCompose(get_num),
        # output_processor=MapCompose(return_num)
        # input_processor=Compose(get_num, stop_on_none=False)
    )
    comment_nums = scrapy.Field(
        input_processor=MapCompose(get_num),
        # output_processor=MapCompose(return_num)
        # input_processor=Compose(get_num, stop_on_none=False)
    )
    tags = scrapy.Field(
        output_processor=Join()
    )

    def get_insert_sql(self):
        insert_sql = """
                insert into jobbole(title, url, url_object_id, front_image_url, front_image_path,praise_nums, fav_nums, 
                comment_nums, tags, content)
                VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
                """
        params = (
            self['title'], self['url'], self['url_object_id'], self['front_image_url'], self['front_image_path'],
            self['praise_nums'], self['fav_nums'], self['comment_nums'], self['tags'], self['content']
        )
        return insert_sql, params


class ArticleItemLoader(ItemLoader):
    default_output_processor = TakeFirst()

模块

`import re'
正则匹配模块

# match函数是从字符串开始处匹配
num = re.match(".*(\d)", 'xx')
# 如果上面没有匹配成功, 会出现AttributeError
num = num.group(1)
# 另外int([])会出现TypeError

import hashlib
将字符串转换为 md5字符串, 必须经过utf-8编码.scrapy的值都是unicode编码
from scrapy.loader import ItemLoader
继承scrapy的ItemLoader, 自定义ArticleItemLoader.
from scrapy.loader.processors import TakeFirst, MapCompose, Join, Identity, Compose
一系列scrapy给定的处理器函数类,TakeFirst是获取列表第一个非空值, MapCompose的参数是多个函数, 能将列表中的每个值通过函数处理,并将处理结果汇成列表再进入下一个函数. Join将列表连接成一个字符串, Identity不作处理, Compose的参数是多个函数, 与MapCompose不同, 是将整个列表传入函数处理

get_num函数

jobbole.py中add_xpath()中加上add_num, if判断就多余了
TypeError也有点多余, 懒得改了

def get_num(value):
  num = re.match(r".*?(\d+?)", value)
  try:
     return int(num.group(1))
   except AttributeError:
      return 0

JobboleArticleItem类

定义item的字段和输入输出处理器, 输入输出处理器的作用时候不同
这里有点疑问:文档说输入处理器作用在解析出一个值后立即作用, 而输出处理器则是在整个列表完成后作用.假如我把Compose写在输出处理器里, Compose不是处理整个列表的吗?有点矛盾
注意的是如果scrapy.Field()中有output_processor将会使default_output_processor失效
另外MapCompose()中的函数是不处理空值.如果是空列表, 那么函数将不生效.
在scrapy源码可以看到用了一个for循环调用函数处理列表中的值

            for v in values:
                next_values += arg_to_iter(func(v))

get_insert_sql方法
写入mysql数据库的语句和参数, 会在pipelines.py中用到

ArticleItemLoader类

为item每个字段赋予一个默认的输出处理器

pipelines.py

import pymysql
from twisted.enterprise import adbapi
from scrapy.pipelines.images import ImagesPipeline


class JobboleArticlePipeline(object):
    def process_item(self, item, spider):
        return item


class JobboleMysqlPipeline(object):

    def __init__(self, dbpool):
        self.dbpool = dbpool

    @classmethod
    def from_settings(cls, settings):
        params = dict(
            host=settings['MYSQL_HOST'],
            db=settings['MYSQL_DBNAME'],
            user=settings['MYSQL_USER'],
            passwd=settings['MYSQL_PASSWORD'],
            charset='utf8',
            cursorclass=pymysql.cursors.DictCursor,
            use_unicode=True
        )
        dbpool = adbapi.ConnectionPool('pymysql', **params)
        return cls(dbpool)

    def process_item(self, item, spider):
        query = self.dbpool.runInteraction(self.do_insert, item)
        query.addErrback(self.handle_error, item, spider)

    def do_insert(self, cursor, item):
        insert_sql, params = item.get_insert_sql()
        cursor.execute(insert_sql, params)

    def handle_error(self, failure, item, spider):
        print(failure)


class ArticleImagePipeline(ImagesPipeline):

    def item_completed(self, results, item, info):
        # 注意这里的判断, 可能front_image_url为空
        if 'front_image_url' in item:
            for _, value in results:
                # print(value)
                image_file_path = value['path']
            item['front_image_path'] = image_file_path
        return item

模块

import pymysql
连接和写入数据库的模块

import pymysql
# 连接pymysql
db = pymysql.connect('localhost', 'root', '123456', 'jobbole')
# 使用cursor()方法获取游标
cursor = db.cursor()
# sql插入语句
insert_sql = "insert into jobbole (字段)values('值')"
# 执行插入
try:
  cursor.execute(insert_sql)
  # 确认提交...
  db.commit()
except:
  # 错误就回滚
  cursor.rollback()
# 关闭连接
db.close()

from twisted.enterprise import adbapi
异步, 不清楚, 先背着吧
from scray.pipelines.images import ImagesPipeline
scrapy的图片存储管道, 需要手动添加pillow模块

JobboleArticlePipeline类

自动生成的管道类

JobboleMysqlPipeline类, 自定义异步写入mysql

settings在settings.py中设置
异步连接mysql?

dbpool = adbapi.ConnectionPool('pymysql', **params)

生成实例...

# 执行 __init__(dbpool), 生成实例
return cls(dbpool)

process_item管道处理item的方法

# 异步执行插入操作?
# 不需要db.commit()
query = self. dbpool.runInteraction(self.do_insert, item)

# 看不懂
# 不用返回item?
query.addErrback(self.handle_error, item, spider)

do_insert
cursor参数在ConnectionPool中获得吗?

ArticleImagePipeline

item_completed
参数results, item, info
主要是记录front_image_path

settings.py

通用

ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 1

mysql设置

MYSQL_HOST = '127.0.0.1'
MYSQL_USER = 'root'
MYSQL_DBNAME = 'jobbole'
MYSQL_PASSWORD = '123456'

管道的启用和优先级

数字越低优先级越高, 对应的是Pipelines.py中编写的管道

ITEM_PIPELINES = {
# 'jobbole_article.pipelines.JobboleArticlePipeline': 300,
    'jobbole_article.pipelines.ArticleImagePipeline': 1,
    'jobbole_article.pipelines.JobboleMysqlPipeline': 2,
}

图片存储目录

import os
# 指定图片下载url的item字段
IMAGES_URLS_FIELD = 'front_image_url'
# 图片存储的父目录, 也是settings.py的父目录, __file__是settings.py?
#abspath绝对路径, dirname父目录
image_dir = os.path.abspath(os.path.dirname(__file__))
# 图片存储的文件夹
IMAGES_STORE = os.path.join(image_dir, 'images')

mysql需要用到的命令

# 查看数据库
show databases;
# 查看表格
show tables;
# 创建数据库
create database jobbole;
# 切换数据库
use jobbole;
# 创建表格
create table(
title varchar(200) not null,
url varchar(300) not null,
url_object_id varchar(50) primary key not null,
front_image_url varchar(200),
praise_nums int(11) not null,
fav_nums int(11) not null,
tags varchar(200),
content longtext not null
)
# 查看数据库编码信息
show variables like 'character_set_database';
# 查看表格第一条记录
select * from jobbole limit 1;
# 查看表格记录的数量
select count(title) from jobbole;
# 查看表格的大小
use information_schema
select concat(round(sum(DATA_LENGTH/1024/1024),2),'MB') as data from TABL
ES where table_schema='jobbole' and table_name='jobbole';
# 清空数据表记录
truncate table jobbole;
# 删除一个字段
alter table <tablename> drop column <column_name>;

问题

第一次只爬取了1300多条文章爬虫就终止了, 不清楚具体原因
封面图片数量明显少, 数据库记录9000多条, 图片只有6000多张
封面图片url为空会报错

'fav_nums': 2,
 'front_image_url': [''],
 'praise_nums': 2,
 'tags': '职场 产品经理 程序员 职场',
 'title': '程序员眼里的 PM 有两种：有脑子的和没脑子的。后者占 90％',
 'url': 'http://blog.jobbole.com/92328/',
 'url_object_id': 'f74aa62b6a79fcf8f294173ab52f4459'}
Traceback (most recent call last):
  File "g:\py3env\bole2\venv\lib\site-packages\twisted\internet\defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "g:\py3env\bole2\venv\lib\site-packages\scrapy\pipelines\media.py", line 79, in process_item
    requests = arg_to_iter(self.get_media_requests(item, info))
  File "g:\py3env\bole2\venv\lib\site-packages\scrapy\pipelines\images.py", line 155, in get_media_requests
    return [Request(x) for x in item.get(self.images_urls_field, [])]
  File "g:\py3env\bole2\venv\lib\site-packages\scrapy\pipelines\images.py", line 155, in <listcomp>
    return [Request(x) for x in item.get(self.images_urls_field, [])]
  File "g:\py3env\bole2\venv\lib\site-packages\scrapy\http\request\__init__.py", line 25, in __init__
    self._set_url(url)
  File "g:\py3env\bole2\venv\lib\site-packages\scrapy\http\request\__init__.py", line 62, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url:

文章中如果有emoji表情, 会出现编码错误.
爬取的时候没有设置输出的日志文件
当add_xpath()中path路径提取为空列表时, 输出输入处理器MapCompose()不起作用.
解决办法是在add_xpath参数额外加上处理器

总结

在能理解的基础上看英文文档要比机翻的中文文档好
不能理解可以看看源码
日志输出到文件
如果不能很好的理解每个部分, 那么需要在看完整体后回顾

selenium登录知乎，爬取问答

编码问题

python 中str 和bytes（二进制）的互相转化

因为scray中的response.body是bytes，所以写入文件要转成string

str = 'abc'
# errors 有strick,ignore
byt = str.encode(encoding='utf8', errors='strick')
# bytes->str
str = byt.decode(encodeing='utf8',errors='ignore')

bytes写入文件中要注意的编码

因为在windows中，新文件默认编码是gbk，所以python解释器会用gbk解析网络数据流。此时往往会失败。要在打开文件时指定编码。

with open('c:test.txt', 'w', encoding='utf8') as f:
  f.write(response.body.decode('utf8', errors='ignore'))

base64图片编码

from PIL import image
from io import BytesIO
import base64
img_src = "data:image/jpg;base64,R0lGODdh.."
img_src = img_src.split(',')[1]
img_src = base64.b64encode(img_src)
img = image.open(BytesIO(img_src))
img.show()

爬虫的小技巧

手动构造response

from scrapy.http import HtmlResponse
body = open("example.html").read()
response = HtmlResponse(url='http://example.com', body=body.encode('utf-8'))

爬虫的url的拼接和跟进

def parse(self, response):
    yield {}
    for url in response.xpath().extract():
    yield scrapy.Request(url=response.urljoin(url), callback=self.parse)
//进一步简化,不要for中extract()和response.urljoin
//如果要对提取的Url作处理，url.extract()?
    for url in response.xpath():
        yield response.follow(url, callback=self.parse)

爬虫的日志

scrapy 文档日志
爬虫日志信息的级别和python的是一样，debug，info,warning,error,critical
Spider类自带日志属性

class ZhihuSpider(scrapy.Spider):
  def func(self)
      self.logger.warning('this is a log')

在Spider类外可以

import logging
logging.warning('this is a log')
# 也可以写不同的logger
logger = logging.getlogger('mycustomlogger')
logger.warning('this is a log')

另外在settings.py中可以设置命令行信息输出的级别和输出的日志文件

LOG_FILE = 'dir'
LOG_LEVEL = logging.WARNING
# 命令行
--logfile FILE
--loglevel LEVEL

re匹配不包含字符串

注意(?=)不占匹配位

s = 'sda'
re.match('s(?=d)$', s) # 匹配失败
# 不能匹配s后含da字符串
re.match('s(?!da)', s)

selenium的使用

selenium python api文档

下载浏览器驱动

chrome版本6.0，最新版本会有missing arguments granttype错误

selenium的方法

from selenium import webdriver
driver = webdriver.Chrome(execute_path="驱动所在目录")
# driver.page_source页面源
# selenium等待
from selenium.webdriver.support.ui import WebDriverwait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
# 10是超时时间，until参数是一个函数，这个函数的参数是driver，返回真假
element = WebDriverWait(driver, 10).until(lambda x:x.find_element_by_xpath(
            "//div[@class='SignContainer-switch']/span"))
# 同上，ec是selenium自带的等待函数
 WebDriverWait(driver, 10).until(ec.text_to_be_present_in_element(
            (By.XPATH, "//div[@class='SignContainer-switch']/span"), '注册'）)

整个爬虫代码

settings.py

# -*- coding: utf-8 -*-
import logging
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'zhihu'

SPIDER_MODULES = ['zhihu.spiders']
NEWSPIDER_MODULE = 'zhihu.spiders'
SQL_DATETIME_FORMAT = "%Y-%m-%d %H:%M:%S"
MYSQL_HOST = '127.0.0.1'
MYSQL_DBNAME = 'zhihuSpider'
MYSQL_USER = 'root'
MYSQL_PASSWORD = '123456'

LOG_LEVEL = logging.WARNING
LOG_FILE = 'G:\py3env\bole2\zhihu\zhihu\zhihu_spider.log'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

COOKIES_ENABLED = True

# Override the default request headers:
# 必须
USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0",

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
  'zhihu.pipelines.ZhihuPipeline': 300,
}

zhihu_login.py

# -*- coding: utf-8 -*-
import scrapy
from zhihu.items import ZhihuQuestionItem, ZhihuAnswerItem, ZhihuItem
import re
import json
import datetime
from selenium import webdriver
# 使文本能解析
#from scrapy.selector import Selector
# 用法：Seletor(text=driver.pager_source).css().extract()
# 打开base64编码的图片
#import base64
#from io import BytesIO, StringIO
import logging
# selenium等待加载相关的模块
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec


class ZhihuLoginSpider(scrapy.Spider):

    name = 'zhihu_login'
    allowed_domains = ['www.zhihu.com']
    # start_requests 初始url
    start_urls = ['https://www.zhihu.com/signup?next=%2F']
    # 获取问题答案的api
    start_answer_url = ["https://www.zhihu.com/api/v4/questions/{0}/answers?include=data[*].is_normal,admin_closed_comment,reward_info,is_collapsed,annotation_action,annotation_detail,collapse_reason,is_sticky,collapsed_by,suggest_edit,comment_count,can_comment,content,editable_content,voteup_count,reshipment_settings,comment_permission,created_time,updated_time,review_info,relevant_info,question,excerpt,relationship.is_authorized,is_author,voting,is_thanked,is_nothelp;data[*].mark_infos[*].url;data[*].author.follower_count,badge[*].topics&offset={2}&limit={1}&sort_by=default"]

    def start_requests(self):
        driver = webdriver.Chrome(executable_path='C:/Users/Administrator/Desktop/chromedriver.exe')
        # 打开网址
        driver.get(start_urls[0])
        # 等待登录元素出现，超时10秒
        element = WebDriverWait(driver, 10).until(lambda x:x.find_element_by_xpath(
            "//div[@class='SignContainer-switch']/span"))
        # 点击登录
        element.click()
        # 等待点击后显示“注册”文本
        WebDriverWait(driver, 10).until(ec.text_to_be_present_in_element(
            (By.XPATH, "//div[@class='SignContainer-switch']/span"), '注册'))
        # 模拟输入账号和密码
        driver.find_element_by_css_selector("div.SignFlow-account input").send_keys("你的账号")
        driver.find_element_by_css_selector("div.SignFlow-password input").send_keys("你的宻码")
        driver.find_element_by_css_selector("button.SignFlow-submitButton").click()
        # 等待页面中某个元素加载完成
        WebDriverWait(driver, 10).until(lambda x:x.find_element_by_xpath(
            "//div[@class='GlobalWrite-navTitle']"))
        # 获取cookie
        Cookies = driver.get_cookies()
        cookie_dict = {}
        for cookie in Cookies:
            cookie_dict[cookie['name']] = cookie['value']
        # 关闭驱动 
        driver.close()
        return [scrapy.Request('https://www.zhihu.com/', cookies=cookie_dict, callback=self.parse)]

    def parse(self, response):
        # 获取页面中所有的链接
        all_urls = response.css("a::attr(href)").extract()
        for url in all_urls:
            # 不匹配https://www.zhihu.com/question/13413413/log
            match_obj = re.match('.*zhihu.com/question/(\d+)(/|$)(?!log)', url)
            if match_obj:
                yield scrapy.Request(response.urljoin(url), callback=self.parse_question)
            else:
                yield scrapy.Request(response.urljoin(url), callback=self.parse)

    def parse_question(self, response):
        if "QuestionHeader-title" in response.text:
            match_obj = re.match(".*zhihu.com/question/(\d+)(/|$)", response.url)
            self.logger.warning('Parse function called on {}'.format(response.url))
            if match_obj:
                self.logger.warning('zhihu id is {}'.format(match_obj.group(1)))
                question_id = int(match_obj.group(1))
                item_loader = ZhihuItem(item=ZhihuQuestionItem(), response=response)
                # ::text前不带空格表示直接子节点的文本
                item_loader.add_css("title", "h1.QuestionHeader-title::text")
                item_loader.add_css("content", ".QuestionHeader-detail ::text")
                item_loader.add_value("url", response.url)
                item_loader.add_value("zhihu_id", question_id)
                # 点击查看全部答案和不点击 ，answer_num两个网页提取的css规则不同。
                # 这里将两个css都写上
                item_loader.add_css("answer_num", "h4.List-headerText span ::text")
                item_loader.add_css("answer_num", "a.QuestionMainAction::text")
                item_loader.add_css("comments_num", "div.QuestionHeader-Comment button::text")
                item_loader.add_css("watch_user_num", "strong.NumberBoard-itemValue::text")
                item_loader.add_css("topics", ".QuestionHeader-topics ::text")
                item_loader.add_value("crawl_time", datetime.datetime.now())
                question_item = item_loader.load_item()
        """没用
        else:
            match_obj = re.match(".*zhihu.com/question/(\d+)(/|$)", response.url)
            if match_obj:
                question_id = int(match_obj.group(1))
                item_loader = ItemLoader(item=ZhihuQuestionItem(), response=response)
                item_loader.add_css("title",
                                "//*[id='zh-question-title']/h2/a/text()|//*[@id='zh-question-title']/h2/span/text()")
                item_loader.add_css("content", ".QuestionHeader-detail")
                item_loader.add_value("url", response.url)
                item_loader.add_value("zhihu_id", question_id)
                item_loader.add_css("answer_num", "#zh-question-answer-num::text")
                item_loader.add_css("comment_num", "#zh-question-meta-wrap a[name='addcomment']::text")
                item_loader.add_css("watch_user_num", "//*[@id='zh-question-side-header-wrap']/text()|"
                                                  "//*[@class='zh-question-followers-sidebar]/div/a/strong/text()")
                item_loader.add_css("topics", ".zm-tag-editor-labels a::text")
                question_item = item_loader.load_item()
        """
        # format(*args, **kwargs)
        # print("{1}{程度}{0}".format("开心", "今天", 程度="很")
        # 今天很开心
        yield scrapy.Request(self.start_answer_url[0].format(question_id, 20, 0),
                             callback=self.parse_answer)
        yield question_item

    def parse_answer(self, response):
        # 网页返回的是json字符串，转为字典对象
        ans_json = json.loads(response.text)
        is_end = ans_json["paging"]['is_end']
        next_url = ans_json["paging"]["next"]
        for answer in ans_json["data"]:
            # 用item直接赋值简单，却不能用processor
            answer_item =ZhihuAnswerItem()
            answer_item["zhihu_id"] = answer["id"]
            answer_item["url"] = answer["url"]
            answer_item["question_id"] = answer["question"]["id"]
            answer_item["author_id"] = answer["author"]["id"] if "id" in answer["author"] else None
            answer_item["content"] = answer["content"] if "content" in answer else None
            answer_item["parise_num"] = answer["voteup_count"]
            answer_item["comments_num"] = answer["comment_count"]
            answer_item["create_time"] = answer["created_time"]
            answer_item["update_time"] = answer["updated_time"]
            answer_item["crawl_time"] = datetime.datetime.now()
            yield answer_item
        if not is_end:
            yield scrapy.Request(next_url,  callback=self.parse_answer)

下面图片中就是查看网页中api

图片.png

pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
import pymysql
from twisted.enterprise import adbapi


class ZhihuPipeline(object):

    def __init__(self, dbpool):
        self.dbpool = dbpool
    
    def process_item(self, item, spider):
        query = self.dbpool.runInteraction(self.do_insert_sql, item)
        query.addErrback(self.handle_error, item, spider)

    def do_insert_sql(self, cursor, item):
        insert_sql, params = item.get_insert_sql()
        cursor.execute(insert_sql, params)

    def handle_error(self, failure, item, spider):
        print(failure)

    @classmethod
    def from_settings(cls, settings):
        params = dict(
            host=settings['MYSQL_HOST'],
            db=settings['MYSQL_DBNAME'],
            user=settings['MYSQL_USER'],
            passwd=settings['MYSQL_PASSWORD'],
            charset='utf8',
            cursorclass=pymysql.cursors.DictCursor,
            use_unicode=True,
        )
        dbpool = adbapi.ConnectionPool("pymysql", **params)
        return cls(dbpool)

items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html
import logging
import datetime
import re
import scrapy
from scrapy.loader.processors import TakeFirst, Join, Compose, MapCompose
from scrapy.loader import ItemLoader


# 提取关注数量，回答数量，评论数量文本中的数字
def extract_num(value):
    # 输出日志信息
    logging.warning('this is function extract_num value:{}'.format(value))
    for val in value:
        if val is not None:
            # 去掉数字中的，
            val = ''.join(val.split(','))
            match_obj = re.match(".*?(\d+)", val)
            if match_obj:
                logging.warning('this is one of value:{}'.format(match_obj.group(1)))
                return int(match_obj.group(1))
            break


# 重写ItemLoader，指定默认输出处理器
class ZhihuItem(ItemLoader):
    # 取列表第一个元素
    default_output_processor = TakeFirst()

class ZhihuQuestionItem(scrapy.Item):

    topics = scrapy.Field(
            # 将主题连接
            output_processor=Join(',')
            )
    url = scrapy.Field()
    title = scrapy.Field()
    content = scrapy.Field()
    answer_num = scrapy.Field(
            # 提取数字
            output_processor=Compose(extract_num)
            )
    comments_num = scrapy.Field(
            output_processor=Compose(extract_num)
            )
    # 关注者数量
    watch_user_num = scrapy.Field(
            output_processor=Compose(extract_num)
            )
    zhihu_id = scrapy.Field()
    crawl_time = scrapy.Field()

    def get_insert_sql(self):
        # on duplicate key update col_name=value(col_name)
        insert_sql = """
            insert into zhihu_question(zhihu_id, topics, url, title, content, answer_num, comments_num,
            watch_user_num,  crawl_time
            )
            VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
            on duplicate key update content=values(content), answer_num=values(answer_num), comments_num=values(
            comments_num), watch_user_num=values(watch_user_num)
            """
        # [Failure instance: Traceback: <class 'AttributeError'>: Use item['crawl_time'] = '2018-10-29 19:16:24' to set field value
        # self.crawl_time = datetime.datetime.now()
        # 用get处理相应的键为空的情况
        # 用datetime.datetime.now()返回的值可以插入数据库
        params = (self.get('zhihu_id'), self.get('topics','null'), self.get('url'), self.get('title'), self.get('content','null'), self.get('answer_num',0), self.get('comments_num',0),
                  self.get('watch_user_num',0),  self.get('crawl_time'))
        return insert_sql, params


class ZhihuAnswerItem(scrapy.Item):

    zhihu_id = scrapy.Field()
    url = scrapy.Field()
    question_id = scrapy.Field()
    author_id = scrapy.Field()
    content = scrapy.Field()
    # 赞
    parise_num = scrapy.Field()
    comments_num = scrapy.Field()
    # 创建时间
    create_time = scrapy.Field()
    update_time = scrapy.Field()
    crawl_time = scrapy.Field()

    def get_insert_sql(self):
        insert_sql = """
            insert into zhihu_answer(zhihu_id, url, question_id, author_id, content, parise_num, comments_num,
            create_time, update_time, crawl_time
            )
            VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
            on duplicate key update content=values(content), comments_num=values(comments_num), parise_num=values(
            parise_num), update_time=values(update_time)
            """
        # fromtimestamp方法的将时间戳转为时间元组
        params = (
            self.get("zhihu_id"), self.get("url"), self.get("question_id"), self.get("author_id"), self.get("content"), self.get("parise_num", 0),
            self.get("commennts_num", 0), datetime.datetime.fromtimestamp(self.get("create_time")), datetime.datetime.fromtimestamp(self.get("update_time")), self.get("crawl_time"),
        )
        return insert_sql, params

总结

有些问题会反复的遇到
程序一步一步写，记得加注释
链接先记下
知乎不用selenium的都失效了，看到不用selenium可以登录的请告知

scrapy入门

来源:天涯明月笙的慕课笔记

准备工作

伯乐在线爬取所有文章

安装模块

爬虫结构

基本命令

jobbole.py

模块

JobboleSpider类

items.py

模块

get_num函数

JobboleArticleItem类

ArticleItemLoader类

pipelines.py

模块

JobboleArticlePipeline类

JobboleMysqlPipeline类, 自定义异步写入mysql

ArticleImagePipeline

settings.py

通用

mysql设置

管道的启用和优先级

图片存储目录

mysql需要用到的命令

问题

总结

selenium登录知乎，爬取问答

编码问题

python 中str 和bytes（二进制）的互相转化

bytes写入文件中要注意的编码

base64图片编码

爬虫的小技巧

手动构造response

爬虫的url的拼接和跟进

爬虫的日志

re匹配不包含字符串

selenium的使用

下载浏览器驱动

selenium的方法

整个爬虫代码

settings.py

zhihu_login.py

pipelines.py

items.py

总结