项目环境：
pycharm
windows
python3.7
新建项目
在想要创建的目录下打开命令行
创建命令：scrapy startproject 项目名
接着创建一个spider：
cd 项目名
scrapy genspider 爬虫名 "域名"
在项目下新建一个python文件，运行此文件即可运行爬虫项目，不需要再去命令行了
from scrapy import cmdline
cmdline.execute('scrapy crawl 爬虫名'.split())
使用以上命令创建的文件目录如下：（我爬取的是京东手机商品信息）
image.png
这里解释各个文件所起的作用
scrapy.cfg：项目的总配置文件，通常无须修改。
spider_JDComments：项目的 Python 模块，程序将从此处导入 Python 代码。
items.py：用于定义项目用到的 Item 类。Item 类就是一个 DTO（数据传输对象），通常就是定义 N 个属性，该类需要由开发者来定义。
pipelines.py：项目的管道文件，它负责处理爬取到的信息。该文件需要由开发者编写。
settings.py：项目的配置文件，在该文件中进行项目相关配置。
JD_spider：在该目录下存放项目所需的蜘蛛，蜘蛛负责抓取项目感兴趣的信息。
接下来开始上代码，代码中会有详细注释（#后的英文为scrapy框架自动生成）
items ，这里是创建想要爬取的字段，这里只是命名，操作在JD_spider中
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class SpiderJdcommentsItem(scrapy.Item):
# define the fields for your item here like:
comment = scrapy.Field() #评论评论
star = scrapy.Field() #评论星级
编写爬虫解析文件
# -*- coding: utf-8 -*-
import scrapy
import re
import json
from ..items import SpiderJdcommentsItem #这里注意，引用SpiderJdcommentsItem时用spider_JDComments.spider_JDComments.SpiderJdcommentsItem会报错
class JdSpiderSpider(scrapy.Spider):
name = 'JD_spider' #爬虫名
# allowed_domains = ['list.jd.com'] #允许在此域名内爬取
start_urls = ['https://list.jd.com/list.html?cat=9987,653,655&page=1&sort=sort_commentcount_desc&trans=1&JL=6_0_0#J_main'] #起始爬取的url地址
def parse(self, response):
#爬取每个手机链接
phone_url_list = response.xpath("//*[@id='plist']/ul/li") #xpath解析获取手机链接列表
for temp in phone_url_list:
temp_url = "https:" + temp.xpath("./div/div[4]/a/@href").get() #获取每个手机链接，这里的xpath是接着phone_url_list的写的，可以看到xpath前有个"."
yield scrapy.Request(url=temp_url, callback=self.get_commenturl) #将手机链接传给处理手机详情页的函数（在那里找到评论真实链接）
#因为scrapy是异步爬取的，也就是说他有可能会先爬取页数靠后的手机信息，而页数靠后的手机信息评论少，所以我只让他爬前4页，每页60*10*80,四页也有一万多条，够用了
for i in range(2,5):
next_link = "https://list.jd.com/list.html?cat=9987,653,655&page=%s&sort=sort_commentcount_desc&trans=1&JL=6_0_0#J_main" % i
yield scrapy.Request(next_link, callback=self.parse)
# #获取下一页链接
# next_link = response.xpath("//span[@class='p-num']//a[@class='pn-next']/@href").getall()
# if next_link:
# next_link = next_link[0]
# yield scrapy.Request("https://list.jd.com/" + next_link, callback=self.parse)
def get_commenturl(self, response):
pattern = re.compile('\d+') # 正则表达式匹配链接中的数字串，下面构造评论的url时会用到
number = pattern.findall(response.url)[0]
for i in range(60): # 获取每个手机的50页评论，因为京东只显示100页评论（虽然看着有10多万条）
#好评信息，链接中score=3，中评2，差评1
comment_url = "https://club.jd.com/comment/productPageComments.action?&productId=%s&score=2&sortType=5&page=%s&pageSize=10&isShadowSku=0&rid=0&fold=1" % (number, i)
yield scrapy.Request(url=comment_url, callback=self.detail)
def detail(self, response):
data = json.loads(response.body.decode(response.encoding)) #response.body获得的是byte类型
#scrapy中response.body 与 response.text区别：https://www.cnblogs.com/themost/p/8471953.html
#对response的参数了解链接：https://blog.csdn.net/l1336037686/article/details/78536694
if(data['comments']): #判断评论信息是否存在，有的手机个别评论页数没有评论信息
jdcomment_item = SpiderJdcommentsItem() # 保存评论和星级数据所用
for temp in data['comments']:
if (temp['content'] and temp['score']):
jdcomment_item['comment'] = temp['content']
jdcomment_item['score'] = temp['score']
yield jdcomment_item
附上对response的参数了解链接：https://blog.csdn.net/l1336037686/article/details/78536694
scrapy中response.body 与 response.text区别：https://www.cnblogs.com/themost/p/8471953.html
设置代理，以免被封ip，在middlewares.py最下面添加如下代码（通用，不用改）
class my_useragent(object):
def process_request(self, request, spider):
USER_AGENT_LIST = [
'MSIE (MSIE 6.0; X11; Linux; i686) Opera 7.23',
'Opera/9.20 (Macintosh; Intel Mac OS X; U; en)',
'Opera/9.0 (Macintosh; PPC Mac OS X; U; en)',
'iTunes/9.0.3 (Macintosh; U; Intel Mac OS X 10_6_2; en-ca)',
'Mozilla/4.76 [en_jp] (X11; U; SunOS 5.8 sun4u)',
'iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2)',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20120813 Firefox/16.0',
'Mozilla/4.77 [en] (X11; I; IRIX;64 6.5 IP30)',
'Mozilla/4.8 [en] (X11; U; SunOS; 5.7 sun4u)'
]
agent = random.choice(USER_AGENT_LIST)
request.headers['User_Agent'] = agent
settings.py设置（需要改的地方后面附上了注释）
# -*- coding: utf-8 -*-
# Scrapy settings for spider_JDComments project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'spider_JDComments'
SPIDER_MODULES = ['spider_JDComments.spiders']
NEWSPIDER_MODULE = 'spider_JDComments.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36' #代理
# Obey robots.txt rules
ROBOTSTXT_OBEY = False #是否遵守爬虫协议
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 0.5 #数字越大，爬的越慢
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'spider_JDComments.middlewares.SpiderJdcommentsSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = { #开启代理
'spider_JDComments.middlewares.my_useragent': 543,
}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = { #开启自己定义的popeline
'spider_JDComments.pipelines.SpiderJdcommentsPipeline': 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
保存文件，若果不需要保存在数据库中，则不需开启上一步settings中自己定义的pipeline，也不需下一步中更改pipelines.py文件，可按如下方式保存数据
进入项目文件夹下打开命令行：scrapy crawl 爬虫名 -o test.csv（存储的文件名）
保存数据到mysql中（在pipelines.py中编写）
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
import pymysql
class SpiderJdcommentsPipeline(object):
def __init__(self): #参看文章：https://blog.csdn.net/loner_fang/article/details/81056191
self.conn = pymysql.connect('127.0.0.1', 'root', '123456', 'jd_spider', charset='utf8', use_unicode=True) #从前到后本地ip，用户名，密码，数据库中table名，后两项保证编码正确
self.cursor = self.conn.cursor() #创建游标
def process_item(self, item, spider):
insert_sql = """insert into jd(score, comment) VALUES (%s, %s)""" #插入数据语句
self.cursor.execute(insert_sql, (item['score'], item['comment'])) #插入操作
self.conn.commit() #提交，不进行提交无法保存到数据库
def close_spider(self, spider):
# 关闭游标和连接
self.cursor.close()
self.conn.close()
接下来运行项目就可以实现向mysql中存取数据
最后附上一些操作mysql遇到的问题（使用mysql workbench操作）
看表中信息：
image.png
导出表和清空表：（表导出为csv格式后不规整，需再操作excell，附上链接：https://www.cnblogs.com/zsgyx/p/10452734.html）
image.png
mysql表中一页只有1000条数据，大概在框框位置会有下一页按钮，叫：fetch next frame of records
image.png
mysql常见数据类型：https://blog.csdn.net/qq_42338771/article/details/89880360
导出数据到scv后，所有列的数据是在同一列的，如下：
image.png
选择分列的符号：
image.png
实现分列：
image.png
mysql导出数据时，原本的字符串中有换行的，导出的excell表中会分别存为一行，原因是导出时将\r\n解析了，解决方法：
用命令删除数据库特定表中的\r\n
UPDATE jd_spider.jd_negative SETcomment= REPLACE( REPLACE(comment, CHAR( 10 ) , '' ) , CHAR( 13 ) , '' ) ;
如果执行这条语句报错，Error Code: 1175. You are using safe update mode and you tried to update a table without a WHERE that uses a KEY column To disable safe mode, toggle the option in Preferences -> SQL Editor and reconnect.
使用命令改变数据库模式即可解决：
SET SQL_SAFE_UPDATES = 0
