Scrapy 的环境搭建

找到python3

> which python3
/Users/macroot/virtualenvs/article_spider/bin/python3

生成虚拟环境

virtualenv --python=/Users/macroot/virtualenvs/article_spider/bin/python3 article_spider

进入文件夹

/Users/macroot [macroot@macroots-MacBook-Pro] [0:02]
> cd imooc

生成爬虫项目

/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:02]
> scrapy startproject ArticleSpider

在项目外面创建spider是错误的。删掉

/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:12]
> scrapy genspider jobbole blog.jobbole.com
Created spider 'jobbole' using template 'basic' 
(article_spider) 
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:13]
> ls
ArticleSpider BingSearch    html_start    jobbole.py    py3rex
(article_spider) 
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:13]
> rm -rf jobbole.py

进入目录去创建spider，scrapy会自己放到spider目录下。

/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:14]
> ls
ArticleSpider BingSearch    html_start    py3rex
(article_spider) 
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:14]
> cd ArticleSpider
(article_spider) 
/Users/macroot/imooc/ArticleSpider [macroot@macroots-MacBook-Pro] [0:14]
> scrapy genspider jobbole blog.jobbole.com
Created spider 'jobbole' using template 'basic' in module:
  ArticleSpider.spiders.jobbole
(article_spider)