scrapy作为一个强大的爬虫框架，就不多作介绍。
今天刚入门scrapy，所以做个简单的使用
Scrapy中文文档
Ubuntu安装scrapy
在Ubuntu下安装scrapy，需要的命令为：
pip install zope.interface
pip install twisted
pip install pyopenssl
pip install scrapy
开始
1. 使用scrapy创建文件夹命令：
scrapy startproject doubanTest
命令执行完系统将在当前目录下生成doubanTest文件
2. 看看最后的代码文件目录（开始时创建不存在douban_spider.py，main.py文件）
3. 网页提取的元素需要在items.py定义
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
import scrapy
class DoubantestItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = scrapy.Field() # 抓取电影标题
content = scrapy.Field() # 抓取电影简介
比如我在代码中定义了抓取电影的标题和电影的简介
关于item的使用，官方文档这么描述:
Item用法类似字典dict
4. 在settings.py定义一些防反爬元素
# -*- coding: utf-8 -*-
# Scrapy settings for doubanTest project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# http://doc.scrapy.org/en/latest/topics/settings.html
# http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'doubanTest'
SPIDER_MODULES = ['doubanTest.spiders']
NEWSPIDER_MODULE = 'doubanTest.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36'
FEED_URI = 'doubanmovie.csv' # 将信息存储到doubanmovie.csv文件中
FEED_FORMAT = 'CSV' # 存储的文件类型
5. 在spiders下创建douban_spider.py，编写主要代码
#-*_coding:utf8-*-
import scrapy
from scrapy.spider import Spider
from scrapy.http import Request
from scrapy.selector import Selector
from doubanTest.items import DoubantestItem
class doubanSpider(Spider):
name="doubanSpider" # 爬虫程序名称
start_urls = ['https://movie.douban.com/top250'] # URL访问列表
def parse(self, response):
# print response.body
item = DoubantestItem() # 定义一个Item对象
selector = Selector(response)
Movies = selector.xpath('//div[@class="info"]')
for each in Movies:
title = each.xpath('div[@class="hd"]/a/span/text()').extract() # 获取电影标题
full_title = "".join(title)
content = each.xpath('div[@class="bd"]/p/text()').extract() # 获取电影简介
# item类似dict, 它的key名称必须与items.py里的名称一致
item['title'] = full_title
item['content'] = ';'.join(content).strip()
yield item
6. 在doubanTest目录下创建一个main.py
编写这两句代码，就可以使用pycharm运行了
#-*_coding:utf8-*-
from scrapy import cmdline
cmdline.execute("scrapy crawl doubanSpider".split())
运行main.py得到的结果
ok,就当作scrapy入门例子吧
