Question

Which of the following statements is true for Ridge Regression?

• A. The regularizer in the objective function of Ridge Regression is used to guard against scenarios where the model works well for the test data, but poorly for the training data.
• B. The regularizer of Ridge Regression uses \(L_1\) norm.
• C. Ridge Regression aims to reduce the number of parameters that have negative values.
• D. The regularizer of Ridge Regression may increase the bias of the model, but it helps in reducing the variance in predictions.

Hint:

Ridge Regression's penalty is the squared L2 norm; think about what shrinking weights does to bias and variance.

Answer 1

The Correct Option is D

Step 1: Recall what Ridge Regression's regularizer does.
Ridge Regression adds a penalty term \(\lambda \sum w_i^2\) to the ordinary least squares cost function, where \(w_i\) are the model weights. This is the squared \(L_2\) norm of the weight vector, not the \(L_1\) norm. Adding this penalty discourages the weights from growing too large.

Step 2: Connect the regularizer to bias and variance.
Shrinking the weights makes the model less sensitive to small fluctuations in the training data, which lowers its variance since it will not swing wildly for different training samples. At the same time, shrinking the weights away from their unrestricted least-squares values introduces some systematic error, which raises the bias slightly. This is the classic bias-variance trade-off that regularization exploits to fight overfitting.

Step 3: Check each option against this.
Option (A) describes the opposite of overfitting: overfitting is when a model works well on training data but poorly on test data, not the other way around, so (A) is wrong.
Option (B) is wrong because Ridge Regression's penalty is the \(L_2\) norm (sum of squared weights); the \(L_1\) norm penalty belongs to Lasso Regression, not Ridge.
Option (C) is wrong because Ridge Regression shrinks weights toward zero but almost never sets them exactly to zero, so it does not reduce the count of negative-valued parameters, and that is not even its goal; sparsity and feature elimination is a property of Lasso, not Ridge.
Option (D) correctly states that the regularizer trades a small increase in bias for a reduction in variance, which is exactly why Ridge Regression helps control overfitting.

Final Answer:
Option (D) is the true statement about Ridge Regression.