爬虫入门文章

https://zhuanlan.zhihu.com/p/24669128
https://zhuanlan.zhihu.com/p/24769534
https://zhuanlan.zhihu.com/p/25200262
https://zhuanlan.zhihu.com/p/26257790

userAgent 和 动态IP设置

http://lawtech0902.com/2017/06/11/scrapy-useragent-proxyip/
https://zhuanlan.zhihu.com/p/29733174
https://github.com/hellysmile/fake-useragent

延迟和禁止cookies

https://blkstone.github.io/2016/03/02/crawler-anti-anti-cheat/

PhantomJs 和 selenium 处理Ajax

https://my.oschina.net/lewisgong/blog/872257
https://chaycao.github.io/2016/08/19/Scrapy-Selenium-Phantomjs/

页面解析 Beautiful xpath css.

https://cuiqingcai.com/1319.html

python

lxml安装

https://pypi.org/project/lxml/#files
pip install lxml-4.2.1-cp27-cp27m-win_amd64.whl
https://blog.csdn.net/g1apassz/article/details/46574963
https://blog.csdn.net/acingdreamer/article/details/53348649

pip升级

pip install --upgrade pip

requirements.txt的创建及使用

https://blog.csdn.net/orangleliu/article/details/60958525

python path 引用

https://blog.csdn.net/tony_wong/article/details/18044273

Scrapy安装错误：Microsoft Visual C++ 14.0 is required...

https://blog.csdn.net/nima1994/article/details/74931621?locationNum=10&fps=1

Scrapy shell

https://blog.csdn.net/laoyang360/article/details/52809927
Scrapy运行ImportError: No module named win32api错误
https://blog.csdn.net/u013687632/article/details/57075514

xpath

https://blog.csdn.net/manongpengzai/article/details/77109600

python log

https://blog.csdn.net/chosen0ne/article/details/7319306

scrapy link extrator

https://www.jianshu.com/p/ff9125650697

启动爬虫

进入项目的根目录，执行下列命令启动spider:
scrapy crawl dmoz