1 基础对象与方法

1.1 nltk.text.Text

>>> from nltk.book import *
*** Introductory Examples for the NLTK Book ***
Loading text1, ..., text9 and sent1, ..., sent9
Type the name of the text or sentence to view it.
Type: 'texts()' or 'sents()' to list the materials.
text1: Moby Dick by Herman Melville 1851
text2: Sense and Sensibility by Jane Austen 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by G . K . Chesterton 1908
>>> text1
<Text: Moby Dick by Herman Melville 1851>
>>> type(text1)
<class 'nltk.text.Text'>
>>> dir(text1)
['_CONTEXT_RE', '_COPY_TOKENS', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__len__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '__weakref__', '_context', 'collocations', 'common_contexts', 'concordance', 'count', 'dispersion_plot', 'findall', 'generate', 'index', 'name', 'plot', 'readability', 'similar', 'tokens', 'unicode_repr', 'vocab']

Text.concordance(word)
查找某个词并显示一些上下文

>>> text1.concordance("affection")
Displaying 3 of 3 matches:
oyously assented ; for besides the affection I now felt for Queequeg , he was a
e enough , yet he had a particular affection for his own harpoon , because it w
ing cobbling jobs . Lord ! what an affection all old women have for tinkers . I

Text.similar(word)
搜索与某个词具有相似上下文的单词

>>> text1.similar("monstrous")
imperial subtly impalpable pitiable curious abundant perilous
trustworthy untoward singular lamentable few determined maddens
horrible tyrannical lazy mystifying christian exasperate

Text.common_contexts([word1, word2, ...])
搜索参数中所有word相同的上下文，即word1、word2 ...相同的上下文

>>> text2.common_contexts(["monstrous", "very"])
a_pretty is_pretty a_lucky am_glad be_glad

Text.dispersion_plot([word1, word2, ...])
用离散图表示预料中各个word出现的位置序列表示

>>> text4.dispersion_plot(["citizens", "democracy", "freedom", "duties", "America"])

dispersion_plot

1.2 nltk.probability.FreqDist

自动识别语料库中词汇的频率分布

>>> fdist = FreqDist(text1)
>>> fdist.plot(50, cumulative=True)

FreqDist.plot

1.3 nltk.util.bigrams

词语搭配是指经常一起出现的词序列，为了获取搭配首先需要从文本中提取双连词，使用bigrams就可以实现这个功能

>>> list(bigrams(["more", "is", "said", "than", "done"]))
[('more', 'is'), ('is', 'said'), ('said', 'than'), ('than', 'done')]

除了一些含生僻词的情况，英语文本中的词语搭配基本上是频繁出现的双连词。nltk.text.Text中提供了collocations(self, num=20, window_size=2)方法可以直接从Text文本中提取常出现的词语搭配，如下

>>> text4.collocations()
United States; fellow citizens; four years; years ago; Federal
Government; General Government; American people; Vice President; Old
World; Almighty God; Fellow citizens; Chief Magistrate; Chief Justice;
God bless; every citizen; Indian tribes; public debt; one another;
foreign nations; political parties

2 语料库及其使用

这里主要使用的是nltk.corpus中的语料

2.1 nltk.corpus.reader

corpus语料库阅读器可以高效地访问大量语料库，提供的主要方法如下：

2.2 几个语料库简介

>>> from nltk.corpus import gutenberg        #古腾堡语料库
>>> from nltk.corpus import webtext          #网络语料库
>>> from nltk.corpus import nps_chat         #聊天文本
>>> from nltk.corpus import brown            #布朗语料库
>>> from nltk.corpus import reuters          #路透社语料库
>>> from nltk.corpus import inaugural        #就职演说语料库

fileids()

>>> gutenberg.fileids()                      #获取语料库中的文件
[u'austen-emma.txt', u'austen-persuasion.txt', u'austen-sense.txt', u'bible-kjv.txt', u'blake-poems.txt', u'bryant-stories.txt', u'burgess-busterbrown.txt', u'carroll-alice.txt', u'chesterton-ball.txt', u'chesterton-brown.txt', u'chesterton-thursday.txt', u'edgeworth-parents.txt', u'melville-moby_dick.txt', u'milton-paradise.txt', u'shakespeare-caesar.txt', u'shakespeare-hamlet.txt', u'shakespeare-macbeth.txt', u'whitman-leaves.txt']

categories

>>> brown.categories()
[u'adventure', u'belles_lettres', u'editorial', u'fiction', u'government', u'hobbies', u'humor', u'learned', u'lore', u'mystery', u'news', u'religion', u'reviews', u'romance', u'science_fiction']

words()

>>> nps_chat.words()                        #获取语料库中的所有词汇列表
[u'now', u'im', u'left', u'with', u'this', u'gay', ...]
>>> gutenberg.words(["austen-emma.txt"])    #获取文件中的词汇列表
[u'[', u'Emma', u'by', u'Jane', u'Austen', u'1816', ...]
>>> brown.words(categories="news")          #获取news分类中的词汇列表
[u'The', u'Fulton', u'County', u'Grand', u'Jury', ...]

sents()

>>> inaugural.sents()      #获取语料库中的所有句子列表
[[u'Fellow', u'-', u'Citizens', u'of', u'the', u'Senate', u'and', u'of', u'the', u'House', u'of', u'Representatives', u':'], [u'Among', u'the', u'vicissitudes', u'incident', u'to', u'life', u'no', u'event', u'could', u'have', u'filled', u'me', u'with', u'greater', u'anxieties', u'than', u'that', u'of', u'which', u'the', u'notification', u'was', u'transmitted', u'by', u'your', u'order', u',', u'and', u'received', u'on', u'the', u'14th', u'day', u'of', u'the', u'present', u'month', u'.'], ...]
>>> gutenberg.sents(["shakespeare-macbeth.txt"])
[[u'[', u'The', u'Tragedie', u'of', u'Macbeth', u'by', u'William', u'Shakespeare', u'1603', u']'], [u'Actus', u'Primus', u'.'], ...]
>>> brown.sents(categories=["mystery"])
[[u'There', u'were', u'thirty-eight', u'patients', u'on', u'the', u'bus', u'the', u'morning', u'I', u'left', u'for', u'Hanover', u',', u'most', u'of', u'them', u'disturbed', u'and', u'hallucinating', u'.'], [u'An', u'interne', u',', u'a', u'nurse', u'and', u'two', u'attendants', u'were', u'in', u'charge', u'of', u'us', u'.'], ...]

raw()

>>> for fileid in webtext.fileids():
...     print fileid, webtext.raw(fileid)[:65], "..."    #获取文件的原始内容
...
firefox.txt Cookie Manager: "Don't allow sites that set removed cookies to se ...
grail.txt SCENE 1: [wind] [clop clop clop]
KING ARTHUR: Whoa there!  [clop ...
overheard.txt White guy: So, do you have any plans for this evening?
Asian girl ...
pirates.txt PIRATES OF THE CARRIBEAN: DEAD MAN'S CHEST, by Ted Elliott & Terr ...
singles.txt 25 SEXY MALE, seeks attrac older single lady, for discreet encoun ...
wine.txt Lovely delicate, fragrant Rhone wine. Polished leather and strawb ...

3 条件频率分布

nltk.probability.ConditionalFreqDist是用来研究不同条件下的频率分布的条件频率分布。提供的主要方法如下：

与FreqDist不同的是ConditionalFreqDist以一个配对的列表作为输入，如：

>>> from nltk.corpus import brown
>>> cfd = nltk.ConditionalFreqDist(
...             (genre, word)
...             for genre in brown.categories()
...             for word in brown.words(categories=genre))

我们单独只看brown中两个类别的语料news和romance，将其组织成ConditionalFreqDist的参数（配对的列表），代码如下

>>> genre_word = [(genre, word) for genre in ["news", "romance"] for word in brown.words(categories=genre)]
>>> len(genre_word)
170576
>>> genre_word[:5]
[('news', u'The'), ('news', u'Fulton'), ('news', u'County'), ('news', u'Grand'), ('news', u'Jury')]
>>> genre_word[-5:]
[('romance', u"I'm"), ('romance', u'afraid'), ('romance', u'not'), ('romance', u"''"), ('romance', u'.')]

conditions

>>> cfd = nltk.ConditionalFreqDist(genre_word)
>>> cfd
<ConditionalFreqDist with 2 conditions>
>>> cfd.conditions()
['romance', 'news']

cfd[condition] & cfd[condition][sample]

>>> cfd["news"]
FreqDist({u'the': 5580, u',': 5188, u'.': 4030, u'of': 2849, u'and': 2146, u'to': 2116, u'a': 1993, u'in': 1893, u'for': 943, u'The': 806, ...})
>>> cfd["romance"]
FreqDist({u',': 3899, u'.': 3736, u'the': 2758, u'and': 1776, u'to': 1502, u'a': 1335, u'of': 1186, u'``': 1045, u"''": 1044, u'was': 993, ...})
>>> cfd["romance"]["love"]
32

tabulate

>>> from nltk.corpus import udhr
>>> languages = ["Chickasaw", "English", "German_Deutsch"]
>>> cfd = nltk.ConditionalFreqDist(
...     (lang, len(word))
...     for lang in languages
...     for word in udhr.words(lang+"-Latin1"))
>>> cfd.tabulate()
                 1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18  19  20  21  23
     Chickasaw 411  99  41  68  91  89  77  70  49  33  16  28  45  10   6   4   5   3   2   1   1   1
       English 185 340 358 114 169 117 157 118  80  63  50  12  11   6   1   0   0   0   0   0   0   0
German_Deutsch 171  92 351 103 177 119  97 103  62  58  53  32  27  29  15  14   3   7   5   2   1   0
>>> cfd.tabulate(conditions=["English", "German_Deutsch"], samples=range(10), cumulative=True)
                  0    1    2    3    4    5    6    7    8    9
       English    0  185  525  883  997 1166 1283 1440 1558 1638
German_Deutsch    0  171  263  614  717  894 1013 1110 1213 1275

4 词汇列表语料库

>>> from nltk.corpus import words
>>> from nltk.corpus import stopwords
>>> from nltk.corpus import names
>>> from nltk.corpus import swadesh

words
英语词汇列表

>>> words = words.words()
>>> len(words)
236736
>>> words[:20]
[u'A', u'a', u'aa', u'aal', u'aalii', u'aam', u'Aani', u'aardvark', u'aardwolf', u'Aaron', u'Aaronic', u'Aaronical', u'Aaronite', u'Aaronitic', u'Aaru', u'Ab', u'aba', u'Ababdeh', u'Ababua', u'abac']

stopwords
停用词语语料库

>>> stopwords.words("english")
[u'i', u'me', u'my', u'myself', u'we', u'our', u'ours', u'ourselves', u'you', u'your', u'yours', u'yourself', u'yourselves', u'he', u'him', u'his', u'himself', u'she', u'her', u'hers', u'herself', u'it', u'its', u'itself', u'they', u'them', u'their', u'theirs', u'themselves', u'what', u'which', u'who', u'whom', u'this', u'that', u'these', u'those', u'am', u'is', u'are', u'was', u'were', u'be', u'been', u'being', u'have', u'has', u'had', u'having', u'do', u'does', u'did', u'doing', u'a', u'an', u'the', u'and', u'but', u'if', u'or', u'because', u'as', u'until', u'while', u'of', u'at', u'by', u'for', u'with', u'about', u'against', u'between', u'into', u'through', u'during', u'before', u'after', u'above', u'below', u'to', u'from', u'up', u'down', u'in', u'out', u'on', u'off', u'over', u'under', u'again', u'further', u'then', u'once', u'here', u'there', u'when', u'where', u'why', u'how', u'all', u'any', u'both', u'each', u'few', u'more', u'most', u'other', u'some', u'such', u'no', u'nor', u'not', u'only', u'own', u'same', u'so', u'than', u'too', u'very', u's', u't', u'can', u'will', u'just', u'don', u'should', u'now', u'd', u'll', u'm', u'o', u're', u've', u'y', u'ain', u'aren', u'couldn', u'didn', u'doesn', u'hadn', u'hasn', u'haven', u'isn', u'ma', u'mightn', u'mustn', u'needn', u'shan', u'shouldn', u'wasn', u'weren', u'won', u'wouldn']

names
英语名字语料库

>>> names.words("female.txt")[:10]
[u'Abagael', u'Abagail', u'Abbe', u'Abbey', u'Abbi', u'Abbie', u'Abby', u'Abigael', u'Abigail', u'Abigale']
>>> names.words("male.txt")[:10]
[u'Aamir', u'Aaron', u'Abbey', u'Abbie', u'Abbot', u'Abbott', u'Abby', u'Abdel', u'Abdul', u'Abdulkarim']

swadesh
比较词表，其中包含了几种语言的约200个常用词的列表

>>> swadesh.fileids()
[u'be', u'bg', u'bs', u'ca', u'cs', u'cu', u'de', u'en', u'es', u'fr', u'hr', u'it', u'la', u'mk', u'nl', u'pl', u'pt', u'ro', u'ru', u'sk', u'sl', u'sr', u'sw', u'uk']
>>> swadesh.words("en")[:20]
[u'I', u'you (singular), thou', u'he', u'we', u'you (plural)', u'they', u'this', u'that', u'here', u'there', u'who', u'what', u'where', u'when', u'how', u'not', u'all', u'many', u'some', u'few']

使用entries()方法可以指定多个语言中的同源词，如下所示：

[(u'je', u'I'), (u'tu, vous', u'you (singular), thou'), (u'il', u'he'), (u'nous', u'we'), (u'vous', u'you (plural)'), (u'ils, elles', u'they'), (u'ceci', u'this'), (u'cela', u'that'), (u'ici', u'here'), (u'l\xe0', u'there')]

5 WordNet

WordNet是面向语义的英语词典（主要是英语中的同义词），暂时不做介绍。

6 处理原始文本

6.1 分词并转化为Text对象

这里的分词指的是将原始文本字符串分割成词汇与标点符号组成的列表，直接使用nltk.word_tokenize()，而后调用nltk.Text()就可以将原始文本转化为nltk文本（Text对象）：

>>> from urllib import urlopen
>>> url = "http://www.gutenberg.org/files/2554/2554-0.txt"
>>> raw = urlopen(url).read().decode("utf-8")
>>> raw[:100]
u'\ufeffThe Project Gutenberg EBook of Crime and Punishment, by Fyodor Dostoevsky\r\n\r\nThis eBook is for the '
>>> tokens = word_tokenize(raw)
[u'\ufeffThe', u'Project', u'Gutenberg', u'EBook', u'of', u'Crime', u'and', u'Punishment', u',', u'by', u'Fyodor', u'Dostoevsky', u'This', u'eBook', u'is', u'for', u'the', u'use', u'of', u'anyone']
>>> text = nltk.Text(tokens)      #转化为nltk.Text对象，而后就可以使用Text中的方法
>>> text
<Text: \ufeffThe Project Gutenberg EBook of Crime and Punishment...>
>>> text[:20]
[u'\ufeffThe', u'Project', u'Gutenberg', u'EBook', u'of', u'Crime', u'and', u'Punishment', u',', u'by', u'Fyodor', u'Dostoevsky', u'This', u'eBook', u'is', u'for', u'the', u'use', u'of', u'anyone']

6.2 处理HTML

通过网络爬虫抓取的网页都是HTML格式的，HTML文件中存在大量的标签、js、表单等与我们实际想要的网页内容并不相干。这里我们使用BeautifulSoup来从html文件中提取网页内容，以供我们使用，使用方法如下：

>>> from urllib import urlopen
>>> from bs4 import BeautifulSoup
>>> url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"
>>> html = urlopen(url).read()
>>> html[:60]
'<!doctype html public "-//W3C//DTD HTML 4.0 Transitional//EN'
>>> soup = BeautifulSoup(html, "lxml")
>>> raw = soup.text
>>> raw[:60]
u"\n\nBBC NEWS | Health | Blondes 'to die out in 200 years'\n\n\n\n\n"
>>> tokens = nltk.word_tokenize(raw)
>>> text = nltk.Text(tokens)
>>> text.concordance("gene")
Displaying 7 of 7 matches:
hey say too few people now carry the gene for blondes to last beyond the next
blonde hair is caused by a recessive gene . In order for a child to have blond
 have blonde hair , it must have the gene on both sides of the family in the g
ere is a disadvantage of having that gene or by chance . They do n't disappear
des would disappear is if having the gene was a disadvantage and I do not thin
er's Polio campaign launched in Iraq Gene defect explains high blood pressure
er's Polio campaign launched in Iraq Gene defect explains high blood pressure

6.3 使用正则表达式

语言处理都涉及模式匹配问题，这就需要使用python中的正则表达式模块re。这里就不对re模块进行赘述，详细的re模块介绍请点击python正则表达式。接下来，介绍几个使用re模块进行字符串匹配的例子：

6.3.1 正则表达式简单示例

>>> wsj = sorted(set(nltk.corpus.treebank.words()))
>>> [w for w in wsj if re.search(r'^[0-9]+\.[0-9]+$', w)]    #匹配小数
[u'0.0085', u'0.05', u'0.1', u'0.16', u'0.2', u'0.25', u'0.28', u'0.3', u'0.4', u'0.5', u'0.50', u'0.54', u'0.56', u'0.60', u'0.7', u'0.82', u'0.84', u'0.9', u'0.95', u'0.99', ...]
>>> [w for w in wsj if re.search(r"^[0-9]{4}$", w)]          #匹配四位整数
[u'1614', u'1637', u'1787', u'1901', u'1903', u'1917', u'1925', u'1929', u'1933', u'1934', u'1948', u'1953', u'1955', u'1956', u'1961', u'1965', u'1966', u'1967', u'1968', u'1969', u'1970', u'1971', u'1972', u'1973', u'1975', u'1976', u'1977', u'1979', u'1980', u'1981', u'1982', u'1983', u'1984', u'1985', u'1986', u'1987', u'1988', u'1989', u'1990', u'1991', u'1992', u'1993', u'1994', u'1995', u'1996', u'1997', u'1998', u'1999', u'2000', u'2005', u'2009', u'2017', u'2019', u'2029', u'3057', u'8300']
>>> [w for w in wsj if re.search(r"^[0-9]+-[a-z]{3,5}$", w)] #匹配数字-单词(长度3-5)
[u'10-day', u'10-lap', u'10-year', u'100-share', u'12-point', u'12-year', u'14-hour', u'15-day', u'150-point', u'190-point', u'20-point', u'20-stock', u'21-month', u'237-seat', u'240-page', u'27-year', u'30-day', u'30-point', u'30-share', u'30-year', u'300-day', u'36-day', u'36-store', u'42-year', u'50-state', u'500-stock', u'52-week', u'69-point', u'84-month', u'87-store', u'90-day']
>>> [w for w in wsj if re.search(r"[a-z]{5,}-[a-z]{2,3}-[a-z]{,6}$", w)]
[u'black-and-white', u'bread-and-butter', u'father-in-law', u'machine-gun-toting', u'savings-and-loan']
>>> [w for w in wsj if re.search(r"(ed|ing)$", w)][:20]      #匹配ed或者ing结尾的词
[u'62%-owned', u'Absorbed', u'According', u'Adopting', u'Advanced', u'Advancing', u'Alfred', u'Allied', u'Annualized', u'Anything', u'Arbitrage-related', u'Arbitraging', u'Asked', u'Assuming', u'Atlanta-based', u'Baking', u'Banking', u'Beginning', u'Beijing', u'Being', ...]

6.3.2 使用re.findall()提取字符块

1. 提取两个或者两个以上的元音序列

>>> wsj = sorted(set(nltk.corpus.treebank.words()))
>>> fd = nltk.FreqDist(vs for word in wsj
...                     for vs in re.findall(r"[aeiou]{2,}", word))
>>> fd.items()[:20]
[(u'aa', 3), (u'eo', 39), (u'ei', 86), (u'ee', 217), (u'ea', 476), (u'oui', 6), (u'ao', 6), (u'eu', 18), (u'au', 106), (u'io', 549), (u'ia', 253), (u'ae', 11), (u'ie', 331), (u'iao', 1), (u'iai', 1), (u'uou', 5), (u'ieu', 3), (u'ai', 261), (u'aii', 1), (u'uee', 4)]

2. 去掉英文词内部的元音

英文文本是高度冗余的，忽略掉词内部的元音，也可以轻松阅读。如下示例，正则表达式从左到右依次匹配词首元音序列、词尾元音序列和所有辅音，其他都被忽略。

>>> ptn = re.compile(r"^[AEIOUaeiou]+|[AEIOUaeiou]+$|[^AEIOUaeiou]")
>>> def compress(word):
...     pieces = ptn.findall(word)
...     return "".join(pieces)
...
>>> english_udhr = nltk.corpus.udhr.words("English-Latin1")
>>> nltk.tokenwrap(compress(w) for w in english_udhr[:100])
u'Unvrsl Dclrtn of Hmn Rghts Prmble Whrs rcgntn of the inhrnt dgnty and\nof the eql and inlnble rghts of all mmbrs of the hmn fmly is the fndtn\nof frdm , jstce and pce in the wrld , Whrs dsrgrd and cntmpt fr hmn\nrghts hve rsltd in brbrs acts whch hve outrgd the cnscnce of mnknd ,\nand the advnt of a wrld in whch hmn bngs shll enjy frdm of spch and\nblf and frdm frm fr and wnt hs bn prclmd as the hghst asprtn of the\ncmmn pple , Whrs it is essntl , if'

3. 提取辅音-元音序列

从罗托卡特语词汇中提取所有辅音-元音序列

>>> rotokas_words = nltk.corpus.toolbox.words("rotokas.dic")
>>> cvs = [cv for w in rotokas_words for cv in re.findall(r"[ptksvr][aeiou]", w)]
>>> cfg = nltk.ConditionalFreqDist(cvs)
>>> cfg.tabulate()
    a   e   i   o   u
k 418 148  94 420 173
p  83  31 105  34  51
r 187  63  84  89  79
s   0   0 100   2   1
t  47   8   0 148  37
v  93  27 105  48  49

4. 查找包含某个辅音-元音对应的单词列表

>>> cv_word_pairs = [(cv, w) for w in rotokas_words
...                     for cv in re.findall(r"[ptksvr][aeiou]", w)]
>>> cv_index = nltk.Index(cv_word_pairs)
>>> cv_index["su"]
[u'kasuari']
>>> cv_index["po"]
[u'kaapo', u'kaapopato', u'kaipori', u'kaiporipie', u'kaiporivira', u'kapo', u'kapoa', u'kapokao', u'kapokapo', u'kapokapo', u'kapokapoa', u'kapokapoa', u'kapokapora', u'kapokapora', u'kapokaporo', u'kapokaporo', u'kapokari', u'kapokarito', u'kapokoa', u'kapoo', u'kapooto', u'kapoovira', u'kapopaa', u'kaporo', u'kaporo', u'kaporopa', u'kaporoto', u'kapoto', u'karokaropo', u'karopo', u'kepo', u'kepoi', u'keposi', u'kepoto']

6.4 规范化文本

规范化文本一般指去掉所有的词缀、提取词干或者词形还原的任务等。
其中词干提取（stemming）是抽取词的词干或者词根形式（并不一定能够表达完整的语义，比如将effective处理成effect）。
而词形还原（lemmatization）是把词汇的任意形式还原为一般形式（能够表达完整的语义，比如将一个单词的过去形式转化为一般形式，将drove处理为drive）。

6.4.1 词干提取

在NLTK包中提供集中常用的词干提取接口：Porter stemmer、Lancaster stemmer 和Snowball stemmer。使用示例如下：

nltk.PorterStremmer

>>> from nltk import PorterStemmer
>>> porter_stemmer = PorterStemmer()
>>> porter_stemmer.stem('maximum')
'maximum'
>>> porter_stemmer.stem('presumably')
u'presum'
>>> porter_stemmer.stem('multiply')
u'multipli'
>>> porter_stemmer.stem('provision')
u'provis'
>>> porter_stemmer.stem('owed')
u'owe'

nltk.LancasterStemmer

>>> from nltk import LancasterStemmer
>>> lancaster_stemmer = LancasterStemmer()
>>> lancaster_stemmer.stem('maximum')
'maxim'
>>> lancaster_stemmer.stem('presumably')
'presum'
>>> lancaster_stemmer.stem('multiply')
'multiply'
>>> lancaster_stemmer.stem('provision')
u'provid'
>>> lancaster_stemmer.stem('owed')
'ow'

nltk.SnowballStemmer

>>> from nltk import SnowballStemmer
>>> snowball_stemmer = SnowballStemmer("english")
>>> snowball_stemmer.stem("maximum")
u'maximum'
>>> snowball_stemmer.stem("presumably")
u'presum'
>>> snowball_stemmer.stem("provision")
u'provis'
>>> snowball_stemmer.stem("owed")
u'owe'

一般使用nltk.PorterStremmer

6.4.2 词形还原

NLTK包中提供了Lemmatization词形还原接口，它可以将语言词汇还原为一般形式，在标记词性的前提下效果非常好，示例如下：
nltk.WordNetLemmatizer

>>> from nltk import WordNetLemmatizer
>>> lmtzr = WordNetLemmatizer()
>>> lmtzr.lemmatize("cars")
u'car'
>>> lmtzr.lemmatize("feet")
u'foot'
>>> lmtzr.lemmatize("people")
'people'
>>> lmtzr.lemmatize("fantasized", pos="v")
u'fantasize'