📐 Linear Regression

Linear regression is a supervised learning algorithm used to predict a continuous output variable based on one or more input features.

• It is widely used for prediction, forecasting, and as a baseline model.
• It assumes that the relationship between inputs and output is linear.

$$y = \theta_0 + \theta_1 x$$

When to Use Linear Regression

Linear regression is ideal for:

• Price prediction
• Trend analysis
• Baseline modeling
• Interpretable relationships
• Fast and simple forecasting

It is often used as a baseline before trying more complex models.

Key Assumptions

Linear regression works best when:

• The relationship is approximately linear
• Errors are independent
• Variance of errors is constant
• Residuals are normally distributed

Understanding these assumptions is important for reliable modeling.

---

🧮 Training Set $(x,y)$

Training Set is the Data that is fed to Leaning Algorithm.

The Learning function outputs a hypothesis function

$$X = \begin{bmatrix} 1 & x_1^{(1)} & \dots & x_n^{(1)} \\ 1 & x_1^{(2)} & \dots & x_n^{(2)} \\ \vdots & \vdots & & \vdots \\ 1 & x_1^{(m)} & \dots & x_n^{(m)} \end{bmatrix}$$

• $x$ = inputs
• $y$ = expected output
• $(x,y)$ = Row in Training Set: single training example
• $(Xi,Yi)$ = i-th training example
• $m$ = Number of training examples - rows in training set
• $n$ = Number of features - columns in training set

Example: 🏠 House price

Size (x₁)	Rooms (x₂)	Price (y)
50	1	150
80	2	230
120	3	310

• 📏 $x_1$ → size of house
• 🛏 $x_2$ → number of Rooms
• $m$ = 3 training examples
• $n$ = 2 features

---

💡 Hypothesis $h_\theta(x)$

Function that maps input $x$ to output $y$ is called Hypothesis function.

Supervised learning works like this:

Training Set → Learning Algorithm → Hypothesis Function

The algorithm outputs a function called: h (hypothesis)

• $h$ = hypothesis, trained Algo that can map $Y$ to $Y$

$$h_\theta(x) = \theta^T x$$

Finding $\theta$

Our goal is to find the best values of $\theta$ that minimize prediction error.

1. Single Variable Linear Regression : $h_\theta(x)$

Linear regression is method of finding a Continues linear relationship between Y and X

When there is only one feature, the model is:

$$y = \theta_0 + \theta_1 x$$ $$h_\theta(x) = \theta_0 + \theta_1 x$$

This represents a straight line, where:

• $y$ = is the predicted value
• $x$ = is the input feature
• $\theta_0$ = is the Y intercept
• $\theta_1$ = is the slope of line

Example:

• $\theta_0 = 0$ → line passes through origin

• $\theta_1 = 0$ → horizontal line

  

2. Multi Variate Linear Regression: $h_\theta(x)$

Linear regression with multiple variables

For multiple features:

$$h_\theta(x) = \theta_0 + \theta_1 x_1 + \dots + \theta_n x_n$$

Matrix form:

$$X = \begin{bmatrix} 1 & 50 & 1 \\ 1 & 80 & 2 \\ 1 & 120 & 3 \end{bmatrix}$$

Parameter Vector

$$\theta = \begin{bmatrix} \theta_0 \\ \theta_1 \\ \theta_2 \end{bmatrix}$$

Target Vector

$$y = \begin{bmatrix} 150 \\ 230 \\ 310 \end{bmatrix}$$

This form is computationally efficient.

Calculating Hypothesis Function

Using Closed-form solution:

$\theta = (X^T X)^{-1} X^T y$

• $\theta_0 \approx 20$ : Base price of house
• $\theta_1 \approx 2$ : Price increase per unit size
• $\theta_2 \approx 30$ : Price increase per additional room

Final Hypothesis Function:

$$House Price = 20 + (2) \cdot Size + (30) \cdot Rooms$$

Model Inference

Test For:

• Size = 100
• Rooms = 2

$$Price = 20 + 2(100) + 30(2)$$ $$Price = 20 + 200 + 60 = 280$$

Predicted price = 280

---

💰 Cost Function ($J(\theta)$)

How bad are our guesses?

Goal of the algorithm is to choose $\theta_0$ & $\theta_1$ such that $h(X)$ come close to $Y$.

• Minimize $J(\theta_0, \theta_1)$ thus minimize Error
• Used to measure how well the model performs

Goal is to for a given hypothesis

$$h_\theta(x) = \theta_0 + \theta_1 x$$

Find $\theta$ that minimizes $J(\theta)$

Mean Squared Error Cost Function (MSE)

The squared error works well for regression problems because it:

• Penalizes large errors
• Is mathematically convenient
• Produces a convex function

The cost function is defined as:

$$J(\theta_0, \theta_1) = J(\theta)= \frac{1}{2m} \sum_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)})^2$$

Where:

• $m$ = number of training examples
• $h_\theta(x^{(i)})$ = prediction for given $x$ input
• $y^{(i)}$ = actual value for given input $x$

The objective is to minimize this function.

Plotting Cost Function

One Feature ($x$) and Two Parameters ($\theta_0$, $\theta_1$)

$h= \theta_0 + \theta_1 x$

Parabola

• $J(\theta_0, \theta_1)$ as Y-Axis
• $\theta_1$ as X-Axis

3D Parabola

• $J(\theta_0, \theta_1)$ as Z-Axis

• $\theta_0$ as X-Axis

• $\theta_1$ as Y-Axis

  

Two Feature ($x_1, x_2$) and Three Parameters ($\theta_0$, $\theta_1$, $\theta_2$)

h = $\theta_0 + \theta_1 x_1 + \theta_2 x_2

That lives in 4 dimensions impossible to visualize

J($\theta_0$, $\theta_1$, $\theta_2$) as W-Axis

• $\theta_0$ as X-Axis
• $\theta_1$ as Y-Axis
• $\theta_2$ as Z-Axis

---

Contour Figure ⛰️

Contour plot is seeing surface plot passing through a horizontal 2D clip plane

Each circle represents:

• All points that have the same height = Cost J(θ).

In the contour plot:

• X-axis → $\theta_0$
• Y-axis → $\theta_1$

Why Contour Plot is Circular?

Contour Plot is Circular because Cost function is Convex.

• Circle represent all points that have the same height.
• Smaller circles → smaller cost
• Larger circles → larger cost

Gradient Descent Moves towards the center of the contour plot where cost is minimum.

• It moves perpendicular to the contour lines because that is the direction of steepest descent.

---

🎢 Gradient Descent

Gradient descent is an optimization algorithm used to minimize the cost function by iteratively moving towards the minimum.

That’s just a fancy name for:

Try numbers → see error → improve numbers → repeat.

Works like

• Start somewhere
• Just like going down from a hill.
• Look around and find local minima and keep on going down repeat till find optimal solution.
• Multiple minima can be found

Start somewhere → Take steps downhill → Reach minimum

Algorithm: Single Variant Linear Regression

For feature index j= 0, 1 repeat until convergence

$$\theta_j := \theta_j - \alpha \frac{1}{m} \sum_{i=1}^{m} \left( h_\theta(x^{(i)}) - y^{(i)} \right) x_j^{(i)}$$

For $j = 0,1$

• Simultaneous compute $\theta_0$, $\theta_1$ and store in temp values
• Simultaneous Update $\theta_0$, $\theta_1$

Where:

• $\alpha$ Learning Rate, How big steps we take down hill
• $_j$ is the parameter index
• $:=$ Assignment Operation eg a= a+1
• $=$ Truth Assertion eg a==a

This process is repeated until convergence.

Algo: Multivariate Linear Regression

Steps:

• For feature index j= 0,1,....n repeat until convergence

$$\theta_j := \theta_j - \alpha \frac{1}{m} \sum_{i=1}^{m} \left( h_\theta(x^{(i)}) - y^{(i)} \right) x_j^{(i)}$$

For $j = 0,1,2..n$

• Simultaneous Compute $\theta_0, \theta_1, .... \theta_n$ and store in temp values
• Simultaneous Update $\theta_0, \theta_1, .... \theta_n$

Final Hypothesis Function

$J(\theta) = \frac{1}{2m} \sum_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)})^2$

Which is equivalent to: $J(\theta) = \frac{1}{2m} \sum_{i=1}^{m} (\theta^T x^{(i)} - y^{(i)})^2$

---

Learning Rate ($\alpha$)

Alpha defines rate of Learning

The update rule is:

$$\theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j} J(\theta)$$

Or Simplified

$$\theta := \theta - \alpha \nabla J(\theta)$$

• Small $\alpha$ → slow learning
• Large $\alpha$ → overshooting the minimum
• Proper $\alpha$ → fast convergence

Deciding Learning Rate ($\alpha$)

Plot the cost function, $J(θ)$ over the number of iterations of gradient descent.

If $J(θ)$ decreases every iteration, then you are probably using a good learning rate.

• Smooth Steadily decreasing
• Flattening near minimum
• No wild jumps & no upward trend

       # Good Learning Rate
       
       |
       |\
       | \
       |  \
Cost   |   \
       |    \____
       |
       +----------------
       Iterations

If J(θ) continuously increases then you probably need to decrease $\alpha$.

• Reason: Large $\alpha$ causes overshooting the minimum, leading to divergence or oscillation.
• Fix: Reduce learning rate.

        |
        |      /
        |     /
Cost    |    /
        |   /
        |  /
        +----------------
        Iterations

If J(θ) decreases but very slowly, then you probably need to increase $\alpha$

• Reason: Small $\alpha$ causes slow convergence, taking many iterations to approach the minimum.
• Fix: Increase learning rate.

      Cost
        |
        |\
        | \
        |  \
        |   \
        |    \______
        +----------------
             Iterations

If J(θ) oscillates, then you probably need to reduce $\alpha$ and scale features.

• Reason:
  • Large $\alpha$ leading to oscillation around the minimum.
  • Features are on different scales,
• Fix: Reduce learning rate and apply feature scaling t

    Cost
    |
    |   /\   /\   /\
    |  /  \ /  \ /  \
    | /
    +----------------
    Iterations

Debugging Learning Rate ($\alpha$)

Behavior	Problem	Fix
Cost increases	α too large	Reduce α
Oscillates	α too large	Reduce α + scale
Very slow decrease	α too small	Increase α
No improvement	Features not scaled	Apply scaling

Derivative Term

Derivative terms defines rate of change of Cost function $J(\theta_1)$ wrt $\theta_1$

• At local minima Derivative Term is = 0.

$$\frac{\partial J(\theta)}{\partial \theta} = 0$$

• Derivative term automatically takes small step when it starts to converge towards local minimal. Having a fixed alpha helps

• Derivative term automatically converge $\theta_1$ towards its local minima from both +ve and -ve slopes:

  

---

Feature Scaling

Feature scaling helps to make the cost function more circular, which allows gradient descent to converge faster.

• Make sure feature are on same scale otherwise contour will be skew elliptical.
• Try to get feature into $-1<= xi<= 1$
• Ideally should be withing range $-3<= xi<= 3$

Problem

Difference in scale of features can create skew ellipse in cost function.

• example:
  • Size of house (0-1000) vs Number of rooms (1-10)

A skew ellipse will have a long axis and a short axis.

• Gradient descent will oscillate across the long axis and take a long time to converge to the minimum.

📏 Solution

1. Min-Max Normalization

This is done by subtracting the minimum value of the feature and dividing by the range (max - min).

• Scale features to [0, 1]

$xi = \frac{xi - min(X)}{max(X) - min(X)}$

2. Mean Normalization:

This is done by subtracting the mean of the feature and dividing by the range (max - min).

• Scale features to have mean 0 and range [-3, 3]

$xi = \frac{xi - Avg(X)}{max(X) - min(X)}$

Alternatively

$xi = \frac{xi - \mu}{\sigma}$

Where:

• $\mu$ = mean of feature = Avg(X)
• $\sigma$ = standard deviation = max(X) - min(X)