Nonlinear Regression Models(video1)

definition

Nonlinear Regression is any relationship between an independent variable and a dependent variable which results in a non-linear function modelled data.

difference between Linear Regression & Non-Linear Regression

Linear Regression：

Remark：the is a mapping which satisfies that .(d is the dimension of independent variable)And the name of is basis functions. shown as follow.

Converting Non-Linear Regression to Linear Regression

• Map to to obtain nonlinear features
• Linear regression on and nonlinear features
  Hence，to solve the Non-Linear Regression，we only need to translate to
  which means:the parameter estimation for linear regression is
  
  
  and

Common basis function design

• polynomial：
• Radial basis function
• sigmoid where
• splines
• fourier
• wavelets

Regulariaztion(video2)

definition

A regularization technique is in simple terms a penalty mechanism which applies shrinkage (driving them closer to zero) of coefficient to build a more robust and parsimonious model.

A regularization technique helps in the following main ways:

• Doesn’t assume any particular distribution of the independent variable(对自变量无限制)
• Address Variance-Bias Tradeoffs. Generally will lower the variance from the model(降方差)
• More robust to handle multicollinearity（很多列相关性高）
• Better sparse data (observations即行数、观测值 < features特征值) handling
• Natural feature selection（数据清洗）
• More accurate prediction on new data as it minimizes overfitting on the train data
• Easier interpretation of the output

概念复习：

norm：

Linear Regression：

Linear Regression with Regularization：

R function的处理

从上图不难看出Linear Regression和Linear Regression with Regularization的区别主要是在如何定义R function （即图中的）
example:
(注：是惩罚力度，是一个超参，训练时固定，需要其他机制去训练
当,得到普通线性回归
当,得到)

• LASSO:
• Rigde:
• Elastic Net:(LASSO和Rigde的综合)：
  

三种回归的特性

Ridge Regression Properties：

• Ridge regression shrinks the coefficients and it helps to reduce the model complexity and multi�collinearity.
• Coefficient of parameters can approach to zero but never become zero .

Lasso Regression Properties：

• The lasso is a shrinkage method like ridge, but acts in a nonlinear manner on the outcome .
• regularization can lead to zero coefficients i.e. some of the features are completely neglected for the evaluation of output. So Lasso regression not only helps in reducing over-fitting but it can help us in feature selection.
• In case, Lasso selects at mostvariable before it saturates.
• If there is a group of variables among which the pairwise correlations are very high, then Lasso select one from the group.(高度相似的变量会随机选择，不固定)

Elastic Net Regression Properties：

• Combination of both and and regularization
• part of the penalty generates a sparse model
• part of the penalty
• Remove the limitation of the number of selected variables(可以选到p个）
• Encouraging group effect
• Stabilize theregularization path