Ridge Regression in R

Ridge regression is a regularized regression algorithm that performs L2 regularization that adds an L2 penalty, which equals the square of the magnitude of coefficients. All coefficients are shrunk by the same factor i.e none are eliminated. L2 regularization will not result in sparse models. Ridge regression adds bias to make the estimates reliable approximations to true population values. Ridge regression proceeds by adding a small value k to the diagonal elements of the correlation matrix i.e ridge regression got its name since the diagonal of ones in the correlation matrix are thought to be a ridge.

Here, k is a positive quantity less than 1(usually less than 0.3). The amount of bias in the estimator is given by:

The covariance matrix is given by: 

There exists a value of k for which the Mean Squared Error(MSE i.e variance plus the bias squared) of the ridge estimator is less than least squares estimator. The appropriate value of k depends on the true regression coefficients(that are being estimated)and the optimality of the ridge solution. 

• When lambda = 0, ridge regression equals least squares regression.
• When lambda = infinity, all coefficients shrunk to zero.

Also, the ideal penalty is in between 0 and infinity. Let’s implement Ridge regression in R programming. 

Ridge regression is a classification algorithm that works in part as it doesn’t require unbiased estimators. Ridge regression minimizes the residual sum of squares of predictors in a given model. Ridge regression includes a shrinks the estimate of the coefficients towards zero.