偶看<<孔乙己>>,用爬虫爬出了孔乙己文中三个字的词组.

孔乙己爬虫

代码如下

import jieba
# 导入结巴模块
with open('kongyiji.txt','r', encoding='utf-8') as f:
    kongyiji = f.read()

seg_list = jieba.cut(kongyiji)
words = list(seg_list)

d = {}
for w in words:
    count = d.get(w, 0)
    d[w] = count + 1

keys = d.keys()

word_list = []
for k in keys:
    word = [k, d.get(k)]
    word_list.append(word)

def max(array):
    m = array[0]
    for i in array:
        if m[1] < i[1]:
            m = i
    return m

def sort(array):
    result = []
    for i in range(len(array)):
        m = max(array)
        result.append(m)
        array.remove(m)
    return result

def fliter(array):
    result = []
    for w in array:
        if len(w[0]) >= 3:
            result.append(w)
    return result

sorted_words = sort(word_list)
result = fliter(sorted_words)

for w in result[:]:
    print(w)

结果是(后面的数字是文章中词组出现的次数)

['孔乙己', 33]
['茴香豆', 5]
['十九个', 4]
['不耐烦', 2]
['掌柜的', 2]
['之乎者也', 2]
['怎么样', 2]
['半懂不懂', 2]
['端出去', 1]
['睁大眼睛', 1]
['自此以后', 1]
['免不了', 1]
['叹一口气', 1]
['十多年', 1]
['伸出头', 1]
['这时候', 1]
['不一会', 1]
['坏脾气', 1]
['第二年', 1]
['背地里', 1]
['做点事', 1]
['涨红了脸', 1]
['大半夜', 1]
['一九一九年', 1]
['努着嘴', 1]
['两三天', 1]
['多不多', 1]
['二十多年', 1]
['乱蓬蓬', 1]
['君子固穷', 1]
['十二岁', 1]
['唠唠叨叨', 1]
['赶热闹', 1]
['曲尺形', 1]
['说笑声', 1]
['对柜里', 1]
['看一看', 1]
['读书人', 1]
['替人家', 1]
['干不了', 1]
['缠夹不清', 1]
当然想查找其他的索引要求可以随意添加,技术比较好实现