Scrapy爬虫（1）-知乎

我的博客
第一次用Scrap创建的爬虫，详细记录下
完整代码请访问这里,不过代码可能有所不同

0.思路验证

在创建这个工程前，我先用段代码来检验基本功能可否运行
知乎可以不用登录获取用户信息，对我来说，方便太多了，而且知乎的查看关注页面那里同时显示有个人用户信息所以直接访问：https://www.zhihu.com/people/（token）/following
就可以找到我要的信息了（虽然关注列表只有20个，不过无所谓）

enter description here

import json
from bs4 import BeautifulSoup
import requests

def user(urltoken):
    headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
    url = 'https://www.zhihu.com/people/'+urltoken+'/following'
    html = requests.get(url, headers=headers).text
    soup = BeautifulSoup(html, 'html.parser')
    json_text = soup.body.contents[1].attrs['data-state']
    ob_json = json.loads(json_text)
    followinglist = ob_json['people']['followingByUser'][urltoken]['ids']
    tempset = set(followinglist)
    tempset.remove(None)
    followinglist = list(tempset)
    user_json = ob_json['entities']['users'][urltoken]
    user_info = user_json['headline']
    try:
        school = user_json['educations'][0]['school']['name']
    except:
        school = ''
    try:
        major = user_json['educations'][0]['major']['name']
    except:
        major = ''
    try:
        job = user_json['employments'][0]['job']['name']
    except:
        job = ''
    try:
        company = user_json['employments'][0]['company']['name']
    except:
        company = ''
    try: 
        description = user_json['description']
    except:
        description = ''
    try:
        business = user_json['business']['name']
    except:
        business = ''
    try:
        zhihu_name = user_json['name']
    except:
        zhihu_name = ''
    try:
        location = user_json['locations'][0]['name']
    except:
        location = ''
    gender = user_json['gender']
    if gender == 1:
        gender = '男'
    elif gender == 0:
        gender = '女'
    else:
        gender = '未知'
    uesr_list = [user_info, job, company, description, business, zhihu_name, location, gender, school, major]
    print(uesr_list)
    return followinglist


urltoken = 'sgai'
for urltoken in user(urltoken):
    print(user(urltoken))

呈现的结果部分如下：

['喜欢用数据讲故事。', '数据挖掘工程师', '物流', '有关于我的工作和生活都在微信公众号：一个程序员的日常；会接数据采集、爬虫定制、数据分析相关的单子，请直接私信我。', '互联网', '路人甲', '上海', '男', '', '']
['不追先圣脚步，我行处，皆道路！', '二当家', '游戏公司', '此心光明，亦复何言。我已委托“维权骑士”（<a href="https://link.zhihu.com/?target=http%3A//rightknights.com" class=" external" target="_blank" rel="nofollow noreferrer"><span class="invisible">http://</span><span class="visible">rightknights.com</span><span class="invisible"></span><i class="icon-external"></i></a>）为我的文章进行维权行动', '互联网', '断罪小学赵日天', '深圳/南京/上海', '男', '', '']
['yang-ze-yong-3', 'zhong-wen-40-43', 'ni-ke-ri-xiang-ji', 'miloyip', 'zuo-wen-jie', 'lxghost', 'mukadas-kadir', 'justin-99-9', 'wang-ruo-shan-88', 'zhaoyanbo0098', 'guo-tao-45-48', 'mtjj', 'satanzhangdi', 'wang-hong-hao-99', 'bei-mang-4', 'water-five', 'li-ji-ren', 'he-jing-92-23', 'wei-lan-tian-4', 'yang-da-bao-32']
['个人微信：Maekcurtain', '414632028', 'UI设计交流群', '', '互联网', '莫若', '', '男', '微信公众号', 'imui1060']
['chenqin', 'xia-ji-ji-28', 'yunshu', 'sizhuren', 'sgai', 'hua-sha-94', 'guahu', 'sun-lin-li-72-11', 'luo-pan-57', 'wang-ni-ma-94', 'si-tu-ying-ying-18', 'zheng-gong-si', 'cao-rui-ting-18', 'tian-ji-shun', 'ding-xiang-yi-sheng', 'jue-qiang-de-nu-li', 'ma-bo-yong', 'xiaoxueli', 'ai-an-dong-ni-tu-zi-73', 'guo-zi-501']
['孤独享受者', '', '', '愿你梦里有喝不完的酒', '', '奥LiVia', '北京', '女', '', '']
['metrodatateam', 'jllijlli', 'zao-meng-zhe-62-62', 'kaiserwang730', 'olivia-60-10', 'qi-e-chi-he-zhi-nan', 'fandaidai', 'an-cheng-98', 'zhou-zuo', 'yang-ru-55-52', 'wang-tiao-tiao-91', 'EDASP', 'ma-ke-28', 'shirley-shan-63', 'lens-27', 'mo-zhi-xian-sheng', 'hu-yang-zi', 'tu-si-ji-da-lao-ye', 'summer-world', 'liusonglin']
['非正经演绎派厨艺新鲜人，公众号：餐桌奇谈', '吃货担当', '美食圈', '我有一颗馋嘴痣～~所有文章及答案均需付费转载，不允许擅自搬运~我已委托“维权骑士”（<a href="https://link.zhihu.com/?target=http%3A//rightknights.com" class=" external" target="_blank" rel="nofollow noreferrer"><span class="invisible">http://</span><span class="visible">rightknights.com</span><span class="invisible"></span><i class="icon-external"></i></a>）为我的文章进行维权行动', '互联网', '芊芊呐小桌儿', '北京', '女', '', '']
['feiyucz', 'richard-45-75', 'nusbrant', 'sgai', 'zhou-dong-yu-55-93', 'easteregg', 'cai-lan-80-17', 'zhao-yu-67-63', 'MrBurning', 'zhouzhao', 'excited-vczh', 'justjavac.com', 'mu-se-wan-sheng-ge', 'simona-wen', 'wstcbh', 'BoomberLiu', 'qing-yuan-zi-84', 'cocokele', 'hei-bai-hui-11-79', 'wangxiaofeng']

由此知道我的想法可以运行起来，所以开始创建工程

1.创建工程

整个流程：从起始url中解析出用户信息，然后进入关注者界面和被关注者界面，提取关系用户ID和新的用户链接，将用户信息和关系用户ID存储到MongoDB中，将新的用户链接交给用户信息解析模块，依次类推。完成循环抓取任务

在终端输入

$ scrapy startproject zhihuSpider

之后会在当前目录生成以下结构

zhihuSpider
    |- scrapy.cfg                        项目部署文件
    |- zhihuSpider                     该项目的python模块，可以在这里加入代码
        |- __init__.py                 
        |- items.py                       主要是将爬取的非结构性的数据源提取结构性数据
        |- middlewares.py          
        |- pipelines.py                  将爬取的数据进行持久化存储
        |-  __pycache__  
        |-  settings.py                  配置文件
        |-  spiders                       放置spider代码的目录
            |-  __init__.py 
            |-  __pycache__

2.创建爬虫模块

在终端输入

$ cd zhihuSpider
$ scrapy genspider -t crawl zhihu.com zhihu.com

这里是用scpry的bench的genspider
语法是scrapy genspider[-t template] <name> <domain>
可以用模板来创建spider
之后spider文件夹下面会多出一个zhihu_com.py的文件，里面有段代码为：

    7     name = 'zhihu.com'
    8     allowed_domains = ['zhihu.com']
    9     start_urls = ['http://zhihu.com/']

其中
name是定义spider名字的字符串，它是必须且唯一的
allowed_domains是可选的，它包含了spider允许爬取的域名列表。当OffsiteMiddleware组件启用是，域名不在列表中的url不会被跟进
start_urls为url列表，当没有使用start_requests()方法配置Requests时，Spider将从该列表中开始进行爬取
重新回到开头看到

class ZhihuComSpider(CrawlSpider):

Spider有3大类，最基本的是Spieder，他的属性有name, allowed_domains, start_urls,custom_settings,crawler,start_requests().除了Spieder外还有CrawlSpider和XMLFeedSpider。
CraelCpider除了从Spider继承过来的属性外，还提供了一个新的属性rules，rules是一个包含一个或多个Rule对象的集合，每个Rule对爬取网站的动作定义了特定的规则。如果多个Rule匹配了相同的链接，则根据它们在rules属性中被定义的顺序，第一个会被使用。
Rule类的原型为：

scrapy.contrib,spiders.Rule(link_extractor,callback=None,cb_kwargs=None,follow=None,process_links=None, process_request=None)

参数说明

link_extractor 是一个LinkExtractor对象，定义了如何从爬取到的页面提取链接。
callback回调函数接受一个response作为一个参数，应避免使用parse作为回调函数
cb_kwargs包含传递给回调函数的参数的字典
follow是一个布尔值，指定了根据规则从respose提取的链接是否需要跟进

然而，知乎跳转到关注人的链接不是完整的，而是类似/perople/xxx/following的，crawlSpider没办法识别，所以不能使用rules（或者是我食用crawlSpider方法不对？我都弄了半天了）
了解了这样多，结果我就直接套上去，不使用rules，因为我的链接不是从网页里面提取的，要自己创造的
直接把url注释掉，因为CrawlSpider属于Spider类，所以调用parse解析就好了

class ZhihuComSpider(CrawlSpider):
   name = 'zhihu.com'
   allowed_domains = ['zhihu.com']
   start_urls = ['https://www.zhihu.com/people/sgai/following']


   #rules = (
   #    Rule(LinkExtractor(allow=r'/people/(\w+)/following$', process_value='my_process_value', unique=True, deny_domains=deny), callback='parse_item', follow=True),
   #)


   def parse(self, response):

然后到setting.py下面更改3个数值

请求头
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
关闭robot
ROBOTSTXT_OBEY = False
关闭cookies追踪
COOKIES_ENABLED = False

3.解析网页

解析网页直接照搬我之前测试是否可以运行的代码，并进行了相对应的修改

• 建立一个url存放列表，进行url去重
• 从网页捕获用户urltoken
• tempset部分用户存在没有none存在的情况，捕获错误并pass
• description部分存在有些用户含有html代码，但是我不知道怎么去除只获取中文。。。。
• 把获取的数据封装到item里面
• 判断捉取的用户列表有没有存在False（用户关注数量低于20个就会出现，由于直接返回False，所以直接用if判断）

def parse(self, response):
        deny = []
        html = response.text
        soup = BeautifulSoup(html, 'html.parser')
        token = soup.find("a",{"class":"Tabs-link"})
        pattern = r'e/(.+)/ac'
        urltoken = re.findall(pattern, str(token))[0]
        json_text = soup.body.contents[1].attrs['data-state']
        ob_json = json.loads(json_text)
        followinglist = ob_json['people']['followingByUser'][urltoken]['ids']
        tempset = set(followinglist)
        try:
            tempset.remove(None)
        except:
            pass
        followinglist = list(tempset)
        user_json = ob_json['entities']['users'][urltoken]
        user_info = user_json['headline']
        try:
            school = user_json['educations'][0]['school']['name']
        except:
            school = '该用户尚未填写'
        try:
            major = user_json['educations'][0]['major']['name']
        except:
            major = '该用户尚未填写'
        try:
            job = user_json['employments'][0]['job']['name']
        except:
            job = '该用户尚未填写'
        try:
            company = user_json['employments'][0]['company']['name']
        except:
            company = '该用户尚未填写'
        try: 
            description = user_json['description']
        except:
            description = '该用户尚未填写'
        try:
            business = user_json['business']['name']
        except:
            business = '该用户尚未填写'
        try:
            zhihu_name = user_json['name']
        except:
            zhihu_name = '该用户尚未填写'
        try:
            location = user_json['locations'][0]['name']
        except:
            location = '该用户尚未填写'
        gender = user_json['gender']
        if gender == 1:
            gender = '男'
        elif gender == 0:
            gender = '女'
        else:
            gender = '未知'

        item =UserInfoItem(urltoken=urltoken,user_info=user_info, job=job, company=company, description=description, business=business, zhihu_name=zhihu_name, location=location, gender=gender, school=school, major=major) 
        yield item
        #print(followinglist)
        for following in followinglist:
            if following:
                url = 'https://www.zhihu.com/people/'+following+'/following'
            #else:
                #url = 'https://www.zhihu.com/people/'+urltoken+'/following'

            if url in deny:
                pass
            else:
                deny.append(url)    
                yield scrapy.Request(url=url,callback=self.parse)

4.定义item

定义两个item

import scrapy


class UserInfoItem(scrapy.Item):

    #id
    urltoken = scrapy.Field()
    #个人简介
    user_info = scrapy.Field()
    #姓名
    zhihu_name = scrapy.Field()
    #居住地
    location = scrapy.Field()
    #技术领域
    business = scrapy.Field()
    #性别
    gender = scrapy.Field()
    #公司
    company = scrapy.Field()
    #职位
    job = scrapy.Field()
    #学校
    school = scrapy.Field()
    #教育经历
    major = scrapy.Field()
    #简介
    description = scrapy.Field()

5.Pipeline

定义一个Pipeline，让scrapy把数据从item存入到mongodb数据库里面，配套设置在settings.py里面

import pymongo



class ZhihuspiderPipeline(object):

    def __init__(self, mongo_url, mongo_db):
        self.mongo_url = mongo_url
        self.mongo_db = mongo_db

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_url=crawler.settings.get('MONGO_URL'),
            mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
        )

    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_url)
        self.db = self.client[self.mongo_db]

    def close_spider(self, spider):
        self.client.close()
    
    def process_item(self, item, spider):
        self.db.UserInfo.insert(dict(item))
        return item

6.其他补充

6.1.利用middlewares.py实现ip代理

配到设置在settings.py里面

import random



class RandomProxy(object):
    def __init__(self,iplist):
        self.iplist=iplist

    @classmethod
    def from_crawler(cls,crawler):
        return cls(crawler.settings.getlist('IPLIST'))

    def process_request(self, request, spider):
        proxy = random.choice(self.iplist)
        request.meta['proxy'] =proxy

6.2setting设置

请求头添加

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'

激活item和中间件

DOWNLOADER_MIDDLEWARES = {
    'zhihuSpider.middlewares.RandomProxy':543
}

ITEM_PIPELINES = {
    'zhihuSpider.pipelines.ZhihuspiderPipeline': 300,
}

激活随机爬取时间

AUTOTHROTTLE_ENABLED = True
# The initial download delay
AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
AUTOTHROTTLE_MAX_DELAY = 60

MONGODB配置

MONGO_URL = 'mongodb://localhost:27017/'
MONGO_DATABASE='zhihu'

ip代理列表
这个ip我是从免费代理IP拿的，测试一下ip代理是否可行，我的ip代理池捉不到什么有效 ip了- -。。。

IPLIST=["https://14.112.76.235:65309"]

7.结果

用几个数据测试一下是否有保存，要爬取全部的话还要等重新换个ip代理池

enter description here

7.优化与补充

• Scrapy 自定义settings--简化编写爬虫操作--加快爬虫速度
• v2ex讨论《scrapy 抓取速度问题》
  还有scrapyd部署和scrapyd-client部署见上一篇文章