Ridge Regression#

The ridge regression method is similar to the least squares procedure except that it penalizes the sizes of the regression coefficients. Ridge regression is one of the most commonly used methods to overcome data multicollinearity.


Let \((x_1, \ldots, x_p)\) be a vector of input variables and \(y = (y_1, \ldots, y_k)\) be the response. For each \(j=1, \ldots, k\), the ridge regression model has the form similar to the linear regression model [Hoerl70], except that the coefficients are estimated by minimizing a different objective function [James2013]:

\[y_j = \beta_{0j} + \beta_{1j}x_1 + \ldots + \beta_{pj}x_p\]

Here \(x_i\), \(i=1, \ldots, p\), are referred to as independent variables, and \(y_j\) are referred to as dependent variables or responses.

Training Stage#

Let \((x_{11}, \ldots, x_{1p}, y_{11}, \ldots, y_{1k}), \ldots, (x_{n1}, \ldots, x_{np}, y_{n1}, \ldots, y_{nk})\) be a set of training data, \(n \gg p\). The matrix \(X\) of size \(n \times p\) contains observations \(x_ij\), \(i=1, \ldots, n\), \(j=1, \ldots, p\), of independent variables.

For each \(y_j\), \(j = 1, \ldots, k\), the ridge regression estimates \((\beta_{0j}, \beta_{1j}, \ldots, \beta_{pj})\) by minimizing the objective function:

\[\sum_{i=1}^{n} (y_{ij} - \beta_{0j} - \sum_{q=1}^{p} (\beta_{qj}x_{iq}))^2 + \lambda_j \sum_{q=1}^{p} \beta_{qj}^2\]

where \(λ_j \geq 0\) are ridge parameters [Hoerl70], [James2013].

Prediction Stage#

Ridge regression based prediction is done for input vector \((x_1, \ldots, x_p)\) using the equation \(y_j = \beta_{0j} + \beta_{1j}x_1 + \ldots + \beta_{pj}x_p\) for each \(j=1, \ldots, k\).