Cost Function
If we have overfitting from our hypothesis function, we can reduce the weight that some of the terms in our function carry by increasing their cost.
Say we wanted to make the following function more quadratic:
We'll want to eliminate the influence of
Without actually getting rid of these features or changing the form of our hypothesis, we can instead modify our cost function:
We've added two extra terms at the end to inflate the cost of θ3 and θ4. Now, in order for the cost function to get close to zero, we will have to reduce the values of θ3 and θ4 to near zero. This will in turn greatly reduce the values of
in our hypothesis function. As a result, we see that the new hypothesis (depicted by the pink curve) looks like a quadratic function but fits the data better due to the extra small terms
We could also regularize all of our theta parameters in a single summation as:
( 针对 线性回归）
The λ, or lambda, is the regularization parameter. It determines how much the costs of our theta parameters are inflated.
Using the above cost function with the extra summation, we can smooth the output of our hypothesis function to reduce overfitting. If lambda is chosen to be too large, it may smooth out the function too much and cause under-fitting. Hence, what would happen if λ=0 or is too small ? ( overfitting)
来源：coursera 斯坦福 吴恩达 机器学习
