Bias-Variance Dilemma

Ideally, one wants to choose a model that both accurately captures the regularities in its training data, but also generalizes well to unseen data.

When a model performs poorly, the key question is:

Is the problem bias or variance?

• High bias → underfitting
• High variance → overfitting

Our goal is to find the balance between the two.

🔬 Diagnosing Bias vs. Variance

Effect of Polynomial Degree $d$

As degree of polynomial $d$ increases:

📚 Training error : $J_{\text{train}}(\Theta)$

• $J_{\text{train}}(\Theta)$ steadily decreases
• Higher-degree models fit the training data better

📘 Cross-validation error $J_{\text{CV}}(\Theta)$

• $J_{\text{CV}}(\Theta)$ decreases, then increases
• Forms a convex (U-shaped) curve

Low $d$ → High bias
High $d$ → High variance
Middle $d$ → Good balance

This behavior helps us diagnose bias vs. variance.

Practical Diagnostic Rule

Situation	$J_{\text{train}}$	$J_{\text{CV}}$	Diagnosis
Both high and similar	High	High	High bias
Large gap (train low, CV high)	Low	High	High variance
Both low and similar	Low	Low	Good fit

Bias vs Variance Summary

Concept	Meaning	Cause	Effect
High Bias	Model too simple	Too few features	Underfitting
High Variance	Model too complex	Too many features	Overfitting

---

High Bias (Underfitting) 🦎

The model is too simple to capture the underlying pattern of the data

Characteristics:

• Model is too simple
• Fails to capture structure in the data
• Adding more data does not help much

Costs

• $J_{\text{train}}(\Theta)$ is high

• $J_{\text{CV}}(\Theta)$ is also high

Problem:

• Poor training performance

  High error as equation does not cover all dataset

$$J_{\text{train}}(\Theta) \text{ is high}$$

• Poor Test performance

  Fail when new data set introduced

$$J_{\text{CV}}(\Theta) \text{ is high}$$

• The model performs poorly everywhere

$$J_{\text{CV}}(\Theta) \approx J_{\text{train}}(\Theta)$$

Interpretation:

• Adding more data usually does not help much
• Increasing model complexity($d$) may help

---

🪱 High Variance (Overfitting)

model is too complex and starts fitting the training data perfectly

• Model can bend heavily to pass through every training point.

Characteristics:

• Model is too complex
• Fits noise in the training data
• Adding more data can help reduce variance
• $J_{\text{train}}(\Theta)$ is low, but $J_{\text{CV}}(\Theta)$ is high

Problem:

Low training error ie. good training performance

$$J_{\text{train}}(\Theta) \text{ is low}$$

Poor test performance lead to poor performance on unseen data.

• Poor generalization to new data

$$J_{\text{CV}}(\Theta) \gg J_{\text{train}}(\Theta)$$

Interpretation:

• Model performs very well on training data
• Performs poorly on unseen data
• Large gap between training and validation error

Solutions

1. Use Regularization term to add Penalty for features
2. Reduce model complexity:
   • Reduce Number of Features: Manually select important features
   • Remove irrelevant variables
   • Use automated model selection methods
3. Add more training data to help the model learn the true underlying pattern and reduce overfitting.

---

Key Insight

• Bias is about model simplicity.
• Variance is about model sensitivity to data.

Good model selection is about finding the degree $d$ that minimizes:

$$J_{\text{CV}}(\Theta)$$

while avoiding both underfitting and overfitting.