Scrapy框架
构架图
- Scrapy Engine（引擎模块）
- Scheduler（调度模块）：负责接受引擎发送过来的Request请求，并按照一定的方式进行整理排队、入队，并且在引擎需要时，交换给引擎
- Downloader（下载模块、下载器）：负责下载引擎模块发送的所有Requests请求，并将其获取到的Responses交换给引擎模块，由引擎交给Spider来处理
- Spider（爬虫模块）：负责处理所有Response，从中分析提取数据，获取Item字段需要的数据，并将需要跟进的URL提交给引擎，再次进入Scheduler
- Item Pipline（管道模块）：负责处理Spider中获取的Item，并进行后期处理
1、安装配置
Windows
pip install --upgrade pip
pip install twisted
pip install lxml
pip pywin32
pip install Scrapy
Ubuntu
sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
pip install --upgrade pip
sudo pip install scrapy
2、操作步骤
1、scrapy startproject JobSpider
2、cd JobSpider cd spider
3、scrapy genspider 爬虫程序名 域名
4、编写items.py文件
import scrapy
class DoubanmoviesItem(scrapy.Item):
name = scrapy.Field()
score = scrapy.Field()
intro = scrapy.Field()
info = scrapy.Field()
5、编写爬虫文件
import scrapy
from ..items import DoubanmoviesItem
class RunmoviesSpider(scrapy.Spider):
name = 'runMovies'
allowed_domains = ['movie.douban.com']
def parse(self, response):
item = DoubanmoviesItem()
content_list = response.xpath("//div[@class='article']/ol/li/div[@class='item']/div[@class='info']")
next_link = response.xpath("//div[@class='paginator']/span/a/@href").extract()
print(next_link)
for content in content_list:
name = content.xpath(".//div[@class='hd']/a/span[1]/text()").extract()[0]
score = content.xpath(".//div[@class='bd']/div[@class='star']/span[@class='rating_num']/text()").extract()[0].strip()
intro = content.xpath(".//div[@class='bd']/p[1]/text()").extract()[0].strip()
info = content.xpath(".//div[@class='bd']/p[@class='quote']/span[@class='inq']/text()").extract()[0].strip()
if name and score and intro and info:
print(name, score, intro, info)
item["name"] = name
item["score"] = score
item["intro"] = intro
item["info"] = info
yield item
time.sleep(random.randint(0, 2))
if next_link:
url = "https://movie.douban.com/top250" + next_link[len(next_link) - 1]
print(url)
yield scrapy.Request(url=url, callback=self.parse)
6、编写pipelines.py 文件
import pymysql
class DoubanmoviesPipeline(object):
def __init__(self):
# 连接数据库
self.my_conn = pymysql.connect(
host='localhost',
port=3306,
database='douban',
user='root',
password='',
charset='utf8',
)
self.my_cursor = self.my_conn.cursor()
def process_item(self, item, spider):
insert_sql = "insert into movies(`name`,`score`,`intro`,`info`) value(%s,%s,%s,%s)"
print(item["name"], item["score"], item["intro"], item["info"])
self.my_cursor.execute(insert_sql, [item["name"], item["score"], item["intro"], item["info"]])
self.my_conn.commit()
return item
def close_item(self, spider):
self.my_cursor.close()
self.my_conn.close()
7、设置settings.py 文件
- 设置请求头
DEFAULT_REQUEST_HEADERS = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; rv,2.0.1) Gecko/20100101 Firefox/4.0.1',
'Accept-Language': 'en',
}
- 设置管道文件
ITEM_PIPELINES = {
'doubanMovies.pipelines.DoubanmoviesPipeline': 300,
}
3、细节
1、Item pipeline
可以通过管道处理爬起的数据，在pipelines.py 文件中对传输过来的数据进行筛选
from scrapy.exceptions import DropItem
class PricePipeline(object):
vat_factor = 1
def process_item(self, item, spider):
if item['price']:
if item['price_excludes_vat']:
item['price'] = item['price']
return item
else:
raise DropItem('Missing price in %s'% item)
将item写入JSON文件
import json
class JsonWriterPipeline(object):
def __init__(self):
self.file = open('items.jl', 'wb')
def process_item(self, item, spider):
line = json.dumps(dict(item)) + "\n"
self.file.write(line)
return item
去重：给item进行赋id，若id重复，则清除，否则加入。
from scrapy.exceptions import DropItem
class DuplicatesPipeline(object):
def __init__(self):
self.ids_seen = set()
def process_item(self, item, spider):
if item['id'] in self.ids_seen:
raise DropItem('Duplicate item found: %s' % item)
else:
self.ids_seen.add(item['id'])
return item
2、Link Extractor
引用
from scrapy.contrib.linkextractors import LinkExtractor
参数
- allow (a regular expression (or list of))
- deny (a regular expression (or list of))
- allow_domains(str or list)
- deny_domains(str or list)
- restrict_xpath(str or list)
- attrs(list)
- unique(boolean)
- process_value(callable)
3、Logging
通过scrapy.log 模块使用，必须通过显示调用scrapy.log.start()来开启
- CRITICAL - 严重错误
- ERROR - 一般错误
- WARRING - 警告错误
- INFO - 一般信息
- DEBUG - 调试信息
scrapy.log模块
启动log功能：
scrapy.log.start(logfile=None, loglevel=None, logstdout=None)
记录信息：
scrapy.log.msg(message, level=INFO, spider=None)
scrapy.log.CRITICAL
scrapy.log.ERROR
scrapy.log.WARRING
scrapy.log.INFO
scrapy.log.DEBUG
通过在setting.py中进行以下设置可以被用来配置logging:
1、LOG_ENABLED 默认: True，启用logging
2、LOG_ENCODING 默认: 'utf-8'，logging使用的编码
3、LOG_FILE 默认: None，在当前目录里创建logging输出文件的文件名
4、LOG_LEVEL 默认: 'DEBUG'，log的最低级别
5、LOG_STDOUT 默认: False 如果为 True，进程所有的标准输出(及错误)将会被重定向到log中。例如，执行 print "hello" ，其将会在Scrapy log中显示
4、email
python可以通过smtplib库发送email，scrapy提供了自己的实现。采用了Twisted非阻塞式IO，其避免了对爬虫的非阻塞式IO的影响。
from scrapy.mail import MailSender
mailer = MailSender()
或者可以传递一个Scrapy设置对象，其会参考setting：
mailer = MailSender.from_setting(settings)
mailer.send(to=["someone@example.com"], subject="Some subject", body="Some body", cc=["another@example.com"])
MailSender类
class scrapy.mail.MailSender(smtphost=None, mailfrom=None, smtpuser=None, smtppass=None, smtpport=None)
参数：
-
smtphost (str) – 发送email的SMTP主机(host)。如果忽略，则使用
MAIL_HOST。 -
mailfrom (str) – 用于发送email的地址(address)(填入
From:) 。 如果忽略，则使用MAIL_FROM。 -
smtpuser – SMTP用户。如果忽略,则使用
MAIL_USER。 如果未给定，则将不会进行SMTP认证(authentication)。 - smtppass (str) – SMTP认证的密码
- smtpport (int) – SMTP连接的短裤
- smtptls – 强制使用STARTTLS
- smtpssl (boolean) – 强制使用SSL连接
4、scrapy框架爬取图片
1、编写Item文件
import scrapy
class MyItem(scrapy.Item):
# ... other item fields ...
image_urls = scrapy.Field()
images = scrapy.Field()
2、开启图片管道
ITEM_PIPELINES = {'scrapy.contrib.pipeline.images.ImagesPipeline': 1}
3、设置图片存储信息
#图片存储位置
IMAGES_STORE = '/path/to/valid/dir'
# 90天的图片失效期限
IMAGES_EXPIRES = 90
#缩略图信息
IMAGES_THUMBS = {
'small': (50, 50),
'big': (270, 270),
}
4、编写爬虫文件
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from ..items import TuxingspiderItem
class TuxingSpider(CrawlSpider):
name = 'tuxing'
allowed_domains = ['so.photophoto.cn']
start_urls = ['http://so.photophoto.cn/tag/%E6%B5%B7%E6%8A%A5']
# 下一页连接
link_next_page = LinkExtractor(restrict_xpaths=("//div[@id='page']/a[@class='pagenexton']/img/@src"))
rules = [
Rule(link_next_page, callback='parse_item', follow=True),
]
def parse_item(self, response):
img_url_list = response.xpath("//ul[@id='list']/li/div[@class='libg']")
for img_url in img_url_list:
item = TuxingspiderItem()
name = img_url.xpath(".//div[@class='text']/div[@class='text2']/a/text()")
url = img_url.xpath(".//div[@class='image']/a/img/@src")
item["name"] = name
item["imagesUrls"] = url
yield item
5、编写pipelines文件
import scrapy
from scrapy.contrib.pipeline.images import ImagesPipeline
from scrapy.exceptions import DropItem
class MyImagesPipeline(ImagesPipeline):
def get_media_requests(self, item, info):
for image_url in item['image_urls']:
yield scrapy.Request(image_url)
def item_completed(self, results, item, info):
image_paths = [x['path'] for ok, x in results if ok]
if not image_paths:
raise DropItem("Item contains no images")
item['image_paths'] = image_paths
return item
