Overfitting and Regularization

choose model -> calculate loss function -> numerical optimization
ridge regression improves the loss function definition of linear regression by introducing variance into the formula
L2 penalty $\sum_{i=1}^n(y_i - \beta_0 - \sum_{j =1}^p\beta_jx_{ij})^2 + \lambda \sum_{j=1}^p\beta_j^2$
constraint conditioned optimization problem. Lagrange multiplier
Hyperparameter Optimization
$\lambda \uparrow \,: variance\,\downarrow\, bias\,\uparrow$
its value is usually obtained by test: cross validation

L1 penalty in loss function $Linear: \sum_{i=1}^n(y_i - \beta_0 - \sum_{j =1}^p\beta_jx_{ij})^2 + \lambda \sum_{j=1}^p|\beta_j|$ $Logistic: min log(P) = \sum^n_{i=1}[ - y_ilog(h_{\beta}(x_i)) - (1 - y_i)log(1 - h_\beta(x_i))] + \lambda||\beta||_1$
L1 penalty in loss function $Linear: \sum_{i=1}^n(y_i - \beta_0 - \sum_{j =1}^p\beta_jx_{ij})^2 + \lambda \sum_{j=1}^p\beta_j^2$ $Logistic: min log(P) = \sum^n_{i=1}[ - y_ilog(h_{\beta}(x_i)) - (1 - y_i)log(1 - h_\beta(x_i))] + \lambda||\beta||_2$
Norm1 = (|x1| + |x2|)
Norm2 = sqrt(x1^2 +x2^2)
Norm3 = cub(x1^3 + x2^3)

what is it:
assess how your model result will generalize to another independent dataset
K-fold Cross Validation
classify the dataset into three parts -training dataset, validation set and testing set; then train a model from the training set, try (3) different lambdas to get three new models, calculate (3) errors in the validation set and determine which lambda has the least error. Then use this lambda and the whole dataset (traing, validation) to get a final model.Sometimes this method is biased because the traing set and validation set may have different characteristic distributions, so we k-fold our set and do cross validation on each choice of classification of set and calculate the average error.
model selection with cross validation:
use cross validation method to do hyperparameter tuning
cross validation can only validate your model selection

	p	n
y	true positive	false positive
n	false negative	true negative

in "true positive":
"true" means: you made a correct prediction
"positive" means: what your prediction is
Different metrics(量度) for model evaluation:
precision = tp / (tp + fp) spam
recall = tp / (tp + fn) recall
accuracy = (tp + tn) / all
cyber security: recall is necessary, improve precision as much as possible
if data is really imbalanced (a huge difference in n and p), look at precision and recall but not accuracy because negative can be too many and accuracy should be high by nature)

receiver operating characteristic curve

image.png
false positive rate = number of flase positive / number of real negative
true positive rate = number of true positive / number of real positive
同样的模型同样的data 取不同的threshold得到的rate
classifiter越凸、凹越好（积分与random chance积分的差越大越好）面积[0.5, 1] 面积越大分类越好
special points in ROC space
best case (0, 1)
worst case:(1, 0)
Area under the Curve of ROC (AUC)
AUC value: [0, 1]
The larger the value is, the better classification performance your classifier has.
ROC AUC is the probability a randomly-chosen positive example is ranked more highly than a randomly-chosen negative example (campared to the ground truth).
- 00110
  abcde
- cabde
  10010 AUC= 0.8
- dcabe ACU = 1