当前位置：网站首页>L1, L2 norm

L1, L2 norm

2022-07-19 03:48:00 【elkluh】

We know that the loss norm of linear regression can be expressed by the following formula ：

$E= \left \| ya-f \right \|^{2}$

a Is the coefficient we require ,y It's the forecast ,f It's the target value . The optimization strategy is to make E The smaller the better. .

The purpose of regularization is , Is that the model is not so sensitive to noise , Reduce overfitting , Make the model more generalized .

L2 norm

If we want to use L2 Norm is ridge regression （ridge regression）, Then there are

$E_1 = \left \| ya-f \right \|^{2} + \alpha\left \| a \right \|^2$

The function is to add a second formula , prevent a Too big ,a Too much will make the model very sensitive to noise , Cause over fitting , So for E Regularize , narrow a The value of , Prevent over fitting .

The solution process is ：

$\frac{\partial E_{1}}{\partial a}=0$ （1）

$2y^{T}(ya-f)+2\alpha a=0$ （2）

$(y^{T}y+\alpha I )a = y^{T}f$ （3）

$a = (y^{T}+\alpha I)^{-1}y^{T}f$ （4）

L1 norm

L1 norm : $\sum |a_{i}|$ , The sum of absolute values . be aware L1 Regularization is the sum of the absolute values of weights ,E Is a function with an absolute value symbol , Therefore, it is not completely differentiable .

The objective function is

$E_1= \left \| ya-f \right \|^{2}+\alpha\left \| a \right \|$

L1 Regularization is the solution to sparse values ,ai A very small value in will be approximated to 0. such L1 Regularization will automatically select important features , Remove unimportant features .

summary

a ------> L1 Regularization ------> Make the result smoother

a ------> L2 Regularization ------> Make the result more sparse

On the left is L2 Regularized image , The picture on the right is L1 Regularization . because L2 The image of is a circle , Therefore, it is easy to produce smooth results , and L1 It's a diamond , It is easy to generate intersections on the coordinate axis , Therefore, there will be sparse relaxation . The outermost blue line is to produce the minimum distance , So taking the nearest intersection is the optimal solution .

原网站

版权声明
本文为[elkluh]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/200/202207170006024112.html