1.介绍
对于线性回归，为了增加模型的泛化能力，我们可以采用岭回归(Ridge Regression)而引入正则项。当然，最好的办法还是增加训练样本，之后我们会看到增大样本后的效果。为了方便以后使用，我们这次按照面向对象思想将岭回归模型进行封装(class Ridge_Regression)。
2.样本生成
与之前一样，我们采用sklearn的datasets模块产生数据,我们将其定义在init方法内。
def __init__(
self,
n_samples=50,
n_features=10,
n_informative=2,
n_targets=5,
noise=30,
bias=10,
random_state=1,
x=None,
y=None
):
if (x is None) and (y is None):
self.X, self.y = datasets.make_regression(
n_samples=n_samples,
n_features=n_features,
n_informative=n_informative,
n_targets=n_targets,
noise=noise,
bias=bias,
random_state=random_state
)
self.y = self.y.reshape(n_samples, n_targets)
elif (x is not None) and (y is not None):
self.X = np.array(x)
self.y = np.array(y)
else:
raise Exception("Input is invalid!")
3.数据预处理
将样本打乱顺序，同时分隔数据集。
def preprocess(self, proportion=0.8):
n = self.X.shape[0] # training samples
n_train = int(n * proportion)
permutation = np.random.permutation(n)
self.X, self.y = self.X[permutation], self.y[permutation]
self.X_train, self.y_train = self.X[:n_train, :], self.y[:n_train, :]
self.X_test, self.y_test = self.X[n_train:, :], self.y[n_train:, :]
return self.X_train, self.y_train, self.X_test, self.y_test
4.模型训练
这里关注alpha参数，表示正则系数。alpha越大，正则惩罚越厉害，alpha=0就是普通的线性回归模型。
def train(self, alpha=0.01, fit_intercept=True, max_iter=10000):
self.ridge_regressor = linear_model.Ridge(alpha=alpha, fit_intercept=fit_intercept,
max_iter=max_iter)
self.ridge_regressor.fit(self.X_train, self.y_train)
5.预测与评价
与之前线性回归一样，不多讲。
def predict(self, X):
return self.ridge_regressor.predict(X)
def loss(self, X, y):
return round(sm.mean_squared_error(X, y), 4)
def variance(self, X, y):
return round(sm.explained_variance_score(X, y), 4)
6.结果
我们对alpha取0.01, 0.1, 0.5进行测试。
if __name__ == "__main__":
ridge_regressor = Ridge_Regression()
X_train, y_train, X_test, y_test = ridge_regressor.preprocess(proportion=0.75)
for alpha in [0.01, 0.1, 0.5]:
ridge_regressor.train(alpha=alpha)
y_predict_test = ridge_regressor.predict(X_test)
print("alpha = {}, test loss: {}".format(round(alpha, 2), ridge_regressor.loss(y_test, y_predict_test)))
print("alpha = {}, variance: {}\n".format(round(alpha, 2), ridge_regressor.variance(y_test, y_predict_test)))
输出为：
# n_samples = 50
alpha = 0.01, test loss: 1821.4612
alpha = 0.01, variance: 0.1471
alpha = 0.1, test loss: 1796.6773
alpha = 0.1, variance: 0.1571
alpha = 0.5, test loss: 1701.2975
alpha = 0.5, variance: 0.1966
可以看到，增大alpha，test loss有减少，但是效果并不好，因为我们的variance得分很低，证明方差很大。下面，我们将样本数量从50，增大到5000，看看效果。
# n_samples = 5000
alpha = 0.01, test loss: 905.622
alpha = 0.01, variance: 0.6951
alpha = 0.1, test loss: 905.6208
alpha = 0.1, variance: 0.6951
alpha = 0.5, test loss: 905.6158
alpha = 0.5, variance: 0.6951
可以看到，loss有明显下降，同时variance得分明显提高，这个效果要比ridge regression本身效果要好（也可能是我数据产生的不太好，没太发挥出来ridge regression模型的效果)。但是，增大样本，毫无疑问是提示模型效果的关键手段。
