Scrapy的使用

创建一个Scrapy项目

scrapy startproject myPorject

Scrapy的项目结构

image.png

spiders:编写爬虫的目录

爬虫的编写规则

import scrapy
from tutorial.items import DmozItem

class DmozSpider(scrapy.Spider):
  name = 'dmoz' #爬虫唯一名称
  allowed_domains = ['dmoz-odp.org'] #爬取主路径
  start_urls = [
    'https://dmoz-odp.org/Computers/Programming/Languages/Python/Books/',
    'https://dmoz-odp.org/Computers/Programming/Languages/Python/Resources/'
  ] #需要爬取的路径

  def parse(self, response):#爬取成功后的回调函数
    sel = scrapy.selector.Selector(response)
    sites = sel.xpath('//*[@id="site-list-content"]/div/div[3]')
    items = []
    for site in sites:
      item = DmozItem()
      item['title'] = site.xpath('a/div/text()').extract()
      item['link'] = site.xpath('a/@href').extract()
      item['desc'] = site.xpath('div/text()').extract()
      items.append(item)
    print(items)
    return items

运行你的爬虫

 scrapy crawl name