线性回归的核心代码
from sklearn.linear_model import LinearRegression
reg = LinearRegression()
reg.fit(ages_train,net_worths_train) # 训练回归直线
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-3-331a5f2fde64> in <module>()
1 from sklearn.linear_model import LinearRegression
2 reg = LinearRegression()
----> 3 reg.fit(ages_train,net_worths_train) # 训练回归直线
NameError: name 'ages_train' is not defined
reg.coef_可以获取斜率
reg.intercept_可以获取截距
reg.score(target,data)可以获取r平方分数,这个是用来衡量这个拟合程度的一个变量.
这个数值介于0-1之间,如果是接近于1,意味着越好.
文本学习
# 提取词干的操作.
from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer("english")
for word in word_list:
words = words + ' ' + stemmer.stem(word)
# 获取词袋的操作.
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
bags_of_words = vectorizer.fit(word_data)
bags_of_words = vectorizer.transform(word_data)
from sklearn.feature_extraction.text import CountVectorizer
#语料
corpus = [
'This is the first document.',
'This is the second second document.',
'And the third one.',
'Is this the first document?',
]
#将文本中的词语转换为词频矩阵
vectorizer = CountVectorizer()
#计算个词语出现的次数
X = vectorizer.fit_transform(corpus)
#获取词袋中所有文本关键词
word = vectorizer.get_feature_names()
print word
#查看词频结果
print X.toarray()
from sklearn.feature_extraction.text import TfidfTransformer
#类调用
transformer = TfidfTransformer()
print transformer
#将词频矩阵X统计成TF-IDF值
tfidf = transformer.fit_transform(X)
#查看数据结构 tfidf[i][j]表示i类文本中的tf-idf权重
print tfidf.toarray()
[u'and', u'document', u'first', u'is', u'one', u'second', u'the', u'third', u'this']
[[0 1 1 1 0 0 1 0 1]
[0 1 0 1 0 2 1 0 1]
[1 0 0 0 1 0 1 1 0]
[0 1 1 1 0 0 1 0 1]]
TfidfTransformer(norm=u'l2', smooth_idf=True, sublinear_tf=False,
use_idf=True)
[[ 0. 0.43877674 0.54197657 0.43877674 0. 0.
0.35872874 0. 0.43877674]
[ 0. 0.27230147 0. 0.27230147 0. 0.85322574
0.22262429 0. 0.27230147]
[ 0.55280532 0. 0. 0. 0.55280532 0.
0.28847675 0.55280532 0. ]
[ 0. 0.43877674 0.54197657 0.43877674 0. 0.
0.35872874 0. 0.43877674]]
偏差,方差以及特征选择的状况
高偏差差对训练数据关系很少,是一种过度的简化.
高方差相反.它不能很好的推广到没有见过的情况.(往往会过拟合的状况.)
