调参 - 超参数和网格搜索
- 超参数：算法运行前需要决定的参数
- 模型参数：算法训练中学习到的参数
- y = a * x + b
例如：决策树算法中，max_depth分支最大深度就是超参数；而分支阈值就是模型参数
要寻找既不过拟合，预测准确率又高的超参数需要：（既便宜又好，互相矛盾，取一个中间值，性价比最高）
- 领域知识：了解业务
- 经验数值：默认值
- 遍历搜索：遍历运行并比较大量超参数，挑出最优值
- 网格搜索
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
# 手写数字数据集
digits = datasets.load_digits()
digits
{'data': array([[ 0., 0., 5., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 10., 0., 0.],
[ 0., 0., 0., ..., 16., 9., 0.],
...,
[ 0., 0., 1., ..., 6., 0., 0.],
[ 0., 0., 2., ..., 12., 0., 0.],
[ 0., 0., 10., ..., 12., 1., 0.]]),
'target': array([0, 1, 2, ..., 8, 9, 8]),
'target_names': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),
'images': array([[[ 0., 0., 5., ..., 1., 0., 0.],
[ 0., 0., 13., ..., 15., 5., 0.],
[ 0., 3., 15., ..., 11., 8., 0.],
...,
[ 0., 4., 11., ..., 12., 7., 0.],
[ 0., 2., 14., ..., 12., 0., 0.],
[ 0., 0., 6., ..., 0., 0., 0.]],
[[ 0., 0., 0., ..., 5., 0., 0.],
[ 0., 0., 0., ..., 9., 0., 0.],
[ 0., 0., 3., ..., 6., 0., 0.],
...,
[ 0., 0., 1., ..., 6., 0., 0.],
[ 0., 0., 1., ..., 6., 0., 0.],
[ 0., 0., 0., ..., 10., 0., 0.]],
[[ 0., 0., 0., ..., 12., 0., 0.],
[ 0., 0., 3., ..., 14., 0., 0.],
[ 0., 0., 8., ..., 16., 0., 0.],
...,
[ 0., 9., 16., ..., 0., 0., 0.],
[ 0., 3., 13., ..., 11., 5., 0.],
[ 0., 0., 0., ..., 16., 9., 0.]],
...,
[[ 0., 0., 1., ..., 1., 0., 0.],
[ 0., 0., 13., ..., 2., 1., 0.],
[ 0., 0., 16., ..., 16., 5., 0.],
...,
[ 0., 0., 16., ..., 15., 0., 0.],
[ 0., 0., 15., ..., 16., 0., 0.],
[ 0., 0., 2., ..., 6., 0., 0.]],
[[ 0., 0., 2., ..., 0., 0., 0.],
[ 0., 0., 14., ..., 15., 1., 0.],
[ 0., 4., 16., ..., 16., 7., 0.],
...,
[ 0., 0., 0., ..., 16., 2., 0.],
[ 0., 0., 4., ..., 16., 2., 0.],
[ 0., 0., 5., ..., 12., 0., 0.]],
[[ 0., 0., 10., ..., 1., 0., 0.],
[ 0., 2., 16., ..., 1., 0., 0.],
[ 0., 0., 15., ..., 15., 0., 0.],
...,
[ 0., 4., 16., ..., 16., 6., 0.],
[ 0., 8., 16., ..., 16., 8., 0.],
[ 0., 1., 8., ..., 12., 1., 0.]]]),
'DESCR': "Optical Recognition of Handwritten Digits Data Set\n===================================================\n\nNotes\n-----\nData Set Characteristics:\n :Number of Instances: 5620\n :Number of Attributes: 64\n :Attribute Information: 8x8 image of integer pixels in the range 0..16.\n :Missing Attribute Values: None\n :Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)\n :Date: July; 1998\n\nThis is a copy of the test set of the UCI ML hand-written digits datasets\nhttp://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits\n\nThe data set contains images of hand-written digits: 10 classes where\neach class refers to a digit.\n\nPreprocessing programs made available by NIST were used to extract\nnormalized bitmaps of handwritten digits from a preprinted form. From a\ntotal of 43 people, 30 contributed to the training set and different 13\nto the test set. 32x32 bitmaps are divided into nonoverlapping blocks of\n4x4 and the number of on pixels are counted in each block. This generates\nan input matrix of 8x8 where each element is an integer in the range\n0..16. This reduces dimensionality and gives invariance to small\ndistortions.\n\nFor info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.\nT. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.\nL. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,\n1994.\n\nReferences\n----------\n - C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their\n Applications to Handwritten Digit Recognition, MSc Thesis, Institute of\n Graduate Studies in Science and Engineering, Bogazici University.\n - E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.\n - Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.\n Linear dimensionalityreduction using relevance weighted LDA. School of\n Electrical and Electronic Engineering Nanyang Technological University.\n 2005.\n - Claudio Gentile. A New Approximate Maximal Margin Classification\n Algorithm. NIPS. 2000.\n"}
print(digits.DESCR)
Optical Recognition of Handwritten Digits Data Set
===================================================
Notes
-----
Data Set Characteristics:
:Number of Instances: 5620
:Number of Attributes: 64
:Attribute Information: 8x8 image of integer pixels in the range 0..16.
:Missing Attribute Values: None
:Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)
:Date: July; 1998
This is a copy of the test set of the UCI ML hand-written digits datasets
http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits
The data set contains images of hand-written digits: 10 classes where
each class refers to a digit.
Preprocessing programs made available by NIST were used to extract
normalized bitmaps of handwritten digits from a preprinted form. From a
total of 43 people, 30 contributed to the training set and different 13
to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of
4x4 and the number of on pixels are counted in each block. This generates
an input matrix of 8x8 where each element is an integer in the range
0..16. This reduces dimensionality and gives invariance to small
distortions.
For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.
T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.
L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,
1994.
References
----------
- C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their
Applications to Handwritten Digit Recognition, MSc Thesis, Institute of
Graduate Studies in Science and Engineering, Bogazici University.
- E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.
- Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.
Linear dimensionalityreduction using relevance weighted LDA. School of
Electrical and Electronic Engineering Nanyang Technological University.
2005.
- Claudio Gentile. A New Approximate Maximal Margin Classification
Algorithm. NIPS. 2000.
特征：64个像素
X = digits.data
y = digits.target
X,X.shape
(array([[ 0., 0., 5., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 10., 0., 0.],
[ 0., 0., 0., ..., 16., 9., 0.],
...,
[ 0., 0., 1., ..., 6., 0., 0.],
[ 0., 0., 2., ..., 12., 0., 0.],
[ 0., 0., 10., ..., 12., 1., 0.]]), (1797, 64))
标签：10个数字
y,y.shape
(array([0, 1, 2, ..., 8, 9, 8]), (1797,))
输出一个数字查看
X[666], X[666].shape
(array([ 0., 0., 5., 15., 14., 3., 0., 0., 0., 0., 13., 15., 9.,
15., 2., 0., 0., 4., 16., 12., 0., 10., 6., 0., 0., 8.,
16., 9., 0., 8., 10., 0., 0., 7., 15., 5., 0., 12., 11.,
0., 0., 7., 13., 0., 5., 16., 6., 0., 0., 0., 16., 12.,
15., 13., 1., 0., 0., 0., 6., 16., 12., 2., 0., 0.]), (64,))
num666 = X[666].reshape(8, 8) # 64个像素值转为8*8二维图像数据
num666
array([[ 0., 0., 5., 15., 14., 3., 0., 0.],
[ 0., 0., 13., 15., 9., 15., 2., 0.],
[ 0., 4., 16., 12., 0., 10., 6., 0.],
[ 0., 8., 16., 9., 0., 8., 10., 0.],
[ 0., 7., 15., 5., 0., 12., 11., 0.],
[ 0., 7., 13., 0., 5., 16., 6., 0.],
[ 0., 0., 16., 12., 15., 13., 1., 0.],
[ 0., 0., 6., 16., 12., 2., 0., 0.]])
y[666]
0
plt.figure(figsize=(1, 1))
plt.imshow(
num666, # 数据
cmap='gray', # 配色，gray灰度
# interpolation='lanczos', # 渲染，模糊
)
<matplotlib.image.AxesImage at 0x9e55240>
output_12_1.png
knn训练和预测
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=666)
from sklearn.neighbors import KNeighborsClassifier
knn_clf = KNeighborsClassifier(n_neighbors=5) # 创建空分类器，默认超参数k值为3
knn_clf.fit(X_train, y_train) # 训练
knn_clf.score(X_test, y_test) # 查看准确率
0.9888888888888889
调参，寻找最佳k值
best_k = -1
best_score = 0.0
for i in range(1, 11): # 如果最大k值为最优，需要继续更大的k值循环
# print(i)
knn_clf = KNeighborsClassifier(n_neighbors=i) # 创建空分类器，超参数k值
knn_clf.fit(X_train, y_train) # 训练
score = knn_clf.score(X_test, y_test) # 查看准确率
if best_score < score:
best_k = i
best_score = score
print('best_k = {}'.format(best_k))
print('best_score = {}'.format(best_score))
best_k = 4
best_score = 0.9916666666666667
kNN另一个超参数
weights:uniform/distance
# uniform默认，统一只计算距离，权重一样
# distance,增加权重考虑
距离的权重：一般将距离的倒数作为权重，加入运算
例：设一个未知点距离最近的3个点
红点a，距离为1
蓝点b，距离为3
蓝点c，距离为4
- 根据投票原则，预测未知点为蓝色
- 但红点离未知点最近，应该加权（距离倒数），计算权重后的预测值为：
- 红色：1
- 蓝色：1/3 + 1/4 = 7/12
- 预测未知点为红色
增加一重超参数遍历，距离权重
best_score = 0.0
best_k = -1
best_weight = ''
for j in ['uniform', 'distance']:
for i in range(1, 11): # 如果最大k值为最优，需要继续更大的k值循环
# print(i)
knn_clf = KNeighborsClassifier(n_neighbors=i, weights=j) # 创建空分类器，超参数k值为3
knn_clf.fit(X_train, y_train) # 训练
score = knn_clf.score(X_test, y_test) # 查看准确率
if best_score < score:
best_score = score
best_k = i
best_weight = j
print('best_score = {}'.format(best_score))
print('best_k = {}'.format(best_k))
print('best_weight = {}'.format(best_weight))
best_score = 0.9916666666666667
best_k = 4
best_weight = uniform
不使用uniform预测，考虑权重的准确率
knn_clf = KNeighborsClassifier(n_neighbors=4, weights='distance') # 创建空分类器，超参数k值为4
knn_clf.fit(X_train, y_train) # 训练
score = knn_clf.score(X_test, y_test) # 查看准确率
score
0.9888888888888889
搜索明可夫斯基距离相应的p
明可夫斯基距离包含下面两种
- p=1：曼哈顿距离
- p=2: 欧氏距离
继续增加超参数：搜索不同距离计算方式
p=2/1/...
# p=2默认为欧拉距离，p=1为曼哈顿距离，其他值为相应距离类型
超参数之间可能有相互依赖关系，如参数p只有在weight=distance考虑权重时才有效
best_score = 0.0
best_k = -1
best_weight = 'distance'
best_p = 0
for j in range(1, 6):
for i in range(1, 11): # 如果最大k值为最优，需要继续更大的k值循环
# print(i)
knn_clf = KNeighborsClassifier(n_neighbors=i, weights=best_weight, p=j) # 创建空分类器
knn_clf.fit(X_train, y_train) # 训练
score = knn_clf.score(X_test, y_test) # 查看准确率
if best_score < score:
best_score = score
best_k = i
best_p = j
print('best_weight = {} 的条件下：'.format(best_weight))
print('best_score = {}'.format(best_score))
print('best_k = {}'.format(best_k))
print('best_p = {}'.format(best_p))
best_weight = distance 的条件下：
best_score = 0.9888888888888889
best_k = 5
best_p = 1
上面手动实现的网格搜索方式，可用sklearn内置方法实现
sklearn内封装好的网格搜索方式：Grid Search
# 首先定义要搜索的参数
# 二维数组内嵌套字典
# 每个字典内都是一组网格搜索，标明每个参数的取值范围
# 注意：p值只有在weights=distance时才有意义
param_grid = [
{ # 需遍历10次
'weights': ['uniform'], # 参数取值范围
'n_neighbors': [i for i in range(1, 11)]
# 这里没有p参数
},
{ # 需遍历50次
'weights': ['distance'],
'n_neighbors': [i for i in range(1, 11)],
'p': [i for i in range(1, 6)]
}
]
# 共需遍历60次
knn_clf = KNeighborsClassifier() # 默认参数，创建空分类器
from sklearn.model_selection import GridSearchCV # CV，使用交叉验证方式获得模型正确率
grid_search = GridSearchCV(knn_clf, param_grid) # 网格搜索参数
%%time
grid_search.fit(X_train, y_train) # 网格搜索训练模型，比较耗时，约4分钟
Wall time: 4min 42s
GridSearchCV(cv=None, error_score='raise',
estimator=KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
metric_params=None, n_jobs=1, n_neighbors=5, p=2,
weights='uniform'),
fit_params=None, iid=True, n_jobs=1,
param_grid=[{'weights': ['uniform'], 'n_neighbors': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}, {'weights': ['distance'], 'n_neighbors': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 'p': [1, 2, 3, 4, 5]}],
pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',
scoring=None, verbose=0)
下面这类最后加下划线的方法，都是标识：非用户传入参数而是根据用户传入参数计算出来的参数
# 网格搜索到的最佳分类器参数
grid_search.best_estimator_
KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
metric_params=None, n_jobs=1, n_neighbors=3, p=3,
weights='distance')
# 输出指定过的最佳参数
grid_search.best_params_
{'n_neighbors': 3, 'p': 3, 'weights': 'distance'}
# 网格搜索达到的准确率
grid_search.best_score_
0.9853862212943633
上面网格搜索到的最佳参数相比之前手动完成的网格搜索得到的参数不同，因为sklearn使用的时CV方式（交叉验证）评价精确度，
虽然测试准确率降低，但避免了过拟合，更好
# 将网格搜索训练好的模型赋值给对象，可以和其他训练好的模型一样使用
knn_clf = grid_search.best_estimator_
knn_clf.predict(X_test) # 直接预测结果
knn_clf.score(X_test, y_test) # 准确率
0.9833333333333333
网格搜索耗时很久，verbose参数可以在执行过程中输出一些信息供用户参考，值越大输出信息越多
注意：执行时间很久，如果使用并行运算占用全部cpu内核会导致系统卡死，只用部分内核运算就行
%%time
grid_search = GridSearchCV(knn_clf, param_grid, n_jobs=2, verbose=2)
grid_search.fit(X_train, y_train)
# 供需要搜索60组参数，uniform10组，distance50组
Fitting 3 folds for each of 60 candidates, totalling 180 fits
[Parallel(n_jobs=2)]: Done 37 tasks | elapsed: 56.0s
[Parallel(n_jobs=2)]: Done 158 tasks | elapsed: 3.2min
Wall time: 3min 39s
[Parallel(n_jobs=2)]: Done 180 out of 180 | elapsed: 3.7min finished
