在上一篇Machine Learning文章,我写了如何在网络上简单爬取资源。这次介绍关于爬取资源的处理，即Data Processing;

我思

二、Data Processing

a、中文词频统计及词云可视化

工具为：中文分词jieba模块,jieba是一款优秀的中文分词处理器，简单、方便且开源；python WordCloud 模块,功能齐全，可玩性及展示性较强；

以下为相关代码：

from scipy.misc import imread
import jieba
import jieba.analyse #关键字提取
from os import path
from wordcloud import WordCloud
import matplotlib.pyplot as plt


file=open(r'./art/鹿鼎记.txt',encoding='utf-8',errors='ignore')
url=r'./art/stop_word.txt'
content=file.read()
file_one=[]
try:
    jieba.analyse.set_stop_words(url) # 除去中文停止词库
    tags=jieba.analyse.extract_tags(content,topK=160,withWeight=True) # 获得关键词及其次数，数量为前160
    for tag in tags:
        file_one.append([tag[0],tag[1]*1000]) # 写入file_one 列表内
        print(tag[0]+'\t'+str(tag[1]*1000)) # 显示

finally:
    print('OK')
patch=r'C:\Users\22109\Downloads\字体-方正兰亭黑体.ttf'      #设定词云显示中文类型
dictionary=dict(file_one) # list——dict类型
bg_pic = imread(r'./art/img.png')
wc = WordCloud(
            # 设置字体
            font_path = patch,
            # 设置背景色
            background_color='white',
            # 允许最大词汇
            max_words=200,
            # 词云形状
            mask=bg_pic,
            # 最大号字体
            max_font_size=200,
            )
wc.generate_from_frequencies(dictionary) #引入字典类型
plt.figure() 
plt.imshow(wc) # plt显示
plt.axis('off')

词云显示图：

Figure_1.png

b、统计可视化处理

b-1、直方图比较

import os
import jieba
import jieba.analyse
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import sys
import matplotlib

def file_read(url,n):
    matplotlib.rcParams['font.family'] = 'sans-serif'
    matplotlib.rcParams['font.sans-serif'] = [u'SimHei']
    matplotlib.rcParams['font.size'] = '10'
    content=open(links,'r').read()
    file_one=[]
    try:
        jieba.analyse.set_stop_words('D:\My Documents\Downloads\jinyong\停用词.txt')
        jieba.load_userdict('D:\My Documents\Downloads\jinyong\dict.txt')
        tags = jieba.analyse.extract_tags(content,topK=120,withWeight=True)
        for i in range(len(tags)):
            file_one.append(tags[i])
    finally:
        print 'OK'
    dictionary=pd.DataFrame(file_one).iloc[0:n,:]
    return dictionary
    
width,n=0.4,21
links='D:\My Documents\Downloads\jinyong\诛仙.txt'
dictionary=file_read(links,n)
plt.bar(range(len(dictionary[0])),dictionary[1]*100,width=width,color='rgy')
for i in range(len(dictionary)):
    plt.text(i-width/4*3,dictionary[1][i]*100,dictionary[0][i])
plt.show()

结果如下（以诛仙为例）：

诛仙词频图

可以看出，主角光环十分强大，而且与故事情节较为呼应的是，张小凡及鬼厉的频率几乎相等，即符合诛仙小说的情节变化（小凡黑化）；有意思的是，从数据来看，陆雪琪的出场率大于碧瑶的出场率，若是不懂情节或未看过原著的话，大部分会认为是：陆雪琪为女主角 ；但是，如果爬取碧瑶青云之战之前的数据，估计碧瑶是遥遥领先的。。。

将武侠小说（鹿鼎记）与诛仙对比：
代码如下：

import os
import jieba
import jieba.analyse
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import sys
import matplotlib


def file_read(url,n,width):
    matplotlib.rcParams['font.family'] = 'sans-serif'
    matplotlib.rcParams['font.sans-serif'] = [u'SimHei']
    matplotlib.rcParams['font.size'] = '8'
    long=len(url)
    for j in range(long):
        content=open(url[j],'r').read()
        file_one=[]
        try:
            jieba.analyse.set_stop_words('D:\My Documents\Downloads\jinyong\停用词.txt')
            jieba.load_userdict('D:\My Documents\Downloads\jinyong\dict.txt')
            tags = jieba.analyse.extract_tags(content,topK=120,withWeight=True)
            for i in range(len(tags)):
                file_one.append(tags[i])
        finally:
            print 'ok'
        dictionary=pd.DataFrame(file_one).iloc[0:n,:]
        plt.bar(np.arange(len(dictionary[0]))-width/long*pow(-1,j),dictionary[1]*100,width=width/long)
        for i in range(len(dictionary)):
            plt.text(np.array(i)-width/long*pow(-1,j)-width/4*3,dictionary[1][i]*100,dictionary[0][i])
            
    plt.show()
    
width,n=0.4,21
links='D:\My Documents\Downloads\jinyong\诛仙.txt'
url='D:\My Documents\Downloads\jinyong\鹿鼎记.txt'
urls=[links,url]
file_read(urls,n,width)

可视化处理：

对比图