⏪ Backpropagation Algorithm (BP)

Backpropagation is the algorithm used to minimize the neural network cost function.

Just like gradient descent in linear and logistic regression, our goal is:

$$\min_\Theta J(\Theta)$$

That is, we want to find parameters $\Theta$ that minimize the cost function.

Where

$$J(\Theta) =- \frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K} \left[ y_k^{(i)} \log((h_\Theta(x^{(i)}))_k) + (1 - y_k^{(i)}) \log(1 - (h_\Theta(x^{(i)}))_k) \right] + \frac{\lambda}{2m} \sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}} (\Theta^{(l)}_{j,i})^2$$

Objective

We want to compute the partial derivatives:

$$\frac{\partial}{\partial \Theta_{i,j}^{(l)}} J(\Theta)$$

These derivatives are used in gradient descent to update the parameters.

---

❗ Loss Function ($\delta_j^{(l)}$ )

A loss function measures how wrong your model’s prediction is compared to the actual value. It answers one simple question:

How far off was the prediction? If actual house price = €500,000

The training process tries to minimize this loss.

• The model makes a prediction: Model predicts = €480,000
• The loss function calculates the error : Error = €20,000
• The optimizer adjusts the parameters to reduce that error.

Repeat this thousands of times → model improves.

Error is represented as:

$\delta_j^{(l)} = Predicted - Actual Value$

$\delta_j^{(l)} = 20000$

where $\delta_j^{(l)}$ represents the error of unit $j$ in layer $l$.

More formally:

$$\delta_j^{(l)} = \frac{\partial}{\partial z_j^{(l)}} \text{cost}(t)$$

So:

• $\delta$ is the derivative of the cost with respect to $z$
• It measures how much that unit contributed to the error
• Larger magnitude → steeper slope → more incorrect

Loss function converts error into a number the model can optimize.

⚖️ Loss vs Cost Function

• Loss → error for one example
• Cost → average loss over the dataset

---

Gradient Computation (Backpropagation)

• Forward propagation → computes activations.
• Backpropagation → computes errors ($\delta$ values).
• Errors are propagated from right to left.
• Gradients are accumulated in $\Delta$.
• Regularization is added for non-bias weights.
• Finally, we divide by $m$ to obtain the average gradient.

Backpropagation Algorithm

Given $m$ training set:

$$\{(x^{(1)}, y^{(1)}), \dots, (x^{(m)}, y^{(m)})\}$$

Step 1: 🌱 Initialize Accumulators

Set:

$$\Delta_{i,j}^{(l)} := 0$$

for all $l, i, j$.

This creates matrices of zeros to accumulate gradients.

Step 2: For each training example $t = 1$ to $m$

Backpropagation works per example, and gradients are summed (or averaged) over the dataset.

Example: For two training examples $(x^{(1)}, y^{(1)})$ and $(x^{(2)}, y^{(2)})$

• compute FP for $(x^{(1)}, y^{(1)})$, Compute BP for $(x^{(1)}, y^{(1)})$
• compute FP for $(x^{(2)}, y^{(2)})$, Compute BP for $(x^{(2)}, y^{(2)})$
• Finally Average (or sum) the gradients

2.1 ⏩ Forward Propagation

Set:

$$a^{(1)} := x^{(t)}$$

Compute forward propagation for:

$$l = 2, 3, \dots, L$$

to obtain activations $a^{(l)}$ for any Network layer $l$:

$$z^{(l)} = \Theta^{(l-1)} a^{(l-1)}$$ $$a^{(l)} = g(z^{(l)})$$

Or when look Forward

$$z^{(l+1)} = \Theta^{(l)} a^{(l)}$$ $$a^{(l+1)} = g\left(z^{(l+1)}\right)$$

Where

• $a^{(l)}$ = activations of layer $l$
• $z^{(l)}$ = linear combination before activation
• $\Theta^{(l)}$ = weight matrix between layer $l$ and $l+1$
• $g(\cdot)$ = activation function

2.2 ❗Compute Output Layer Error ($\delta^{(L)}$)

Using the true label $y^{(t)}$:

$$\delta^{(L)} = a^{(L)} - y^{(t)}$$

This is the error of the output layer.

2.3 ⏪ Backpropagate the Errors $\delta^{(L-1)}, \delta^{(L-2)}, \dots \delta^{(2)}$

For layers:

$$l = L-1, L-2, \dots, 2$$

Compute:

$$\delta^{(l)} = \left( (\Theta^{(l)})^T \delta^{(l+1)} \right) \;.\!* \; g'(z^{(l)})$$

For sigmoid activation:

$$g'(z^{(l)}) = a^{(l)} \;.\!* \; (1 - a^{(l)})$$

So equivalently:

$$\delta^{(l)} = \left( (\Theta^{(l)})^T \delta^{(l+1)} \right) \;.\!* \; a^{(l)} \;.\!* \; (1 - a^{(l)})$$

The operator $.\!*$ denotes element-wise multiplication.

2.4 📥 Accumulate Gradients

Update:

$$\Delta_{i,j}^{(l)} := \Delta_{i,j}^{(l)}+ a_j^{(l)} \delta_i^{(l+1)}$$

Vectorized form:

$$\Delta^{(l)} := \Delta^{(l)}+ \delta^{(l+1)} (a^{(l)})^T$$

Step 3: 🎢 Compute Gradients

After processing all training examples:

For $j \ne 0$ (non-bias terms):

$$D_{i,j}^{(l)} = \frac{1}{m} \left( \Delta_{i,j}^{(l)} + \lambda \Theta_{i,j}^{(l)} \right)$$

For bias terms ($j = 0$):

$$D_{i,j}^{(l)} = \frac{1}{m} \Delta_{i,j}^{(l)}$$

Final Result

The gradient of the cost function is:

$$\frac{\partial}{\partial \Theta_{i,j}^{(l)}} J(\Theta) = D_{i,j}^{(l)}$$

The matrix $D^{(l)}$ gives the partial derivatives used in gradient descent.

---

Example:

Given one training example $(x, y)$

Layer 1 (Input)

⏩ Forward Propagation

$$a^{(1)} = x$$

⏪ Backward Propagation

No Error Term Associated with Input Term

---

Layer 2

⏩ Forward Propagation

$$z^{(2)} = \Theta^{(1)} a^{(1)}$$ $$a^{(2)} = g(z^{(2)})$$

(Add bias unit if applicable.)

⏪ Backward Propagation

$$\delta^{(2)} = (\Theta^{(2)T} \delta^{(3)}) \odot g'(z^{(2)})$$

---

Layer 3

⏩ Forward Propagation

$$z^{(3)} = \Theta^{(2)} a^{(2)}$$ $$a^{(3)} = g(z^{(3)})$$

⏪ Backward Propagation

$$\delta^{(3)} = (\Theta^{(3)T} \delta^{(4)}) \odot g'(z^{(3)})$$

---

Layer 4 (Output)

⏩ Forward Propagation

$$z^{(4)} = \Theta^{(3)} a^{(3)}$$ $$a^{(4)} = h_\Theta(x) = g(z^{(4)})$$

⏪ Backward Propagation

Output Layer Error = Calculated Value - Actual Value

$$\delta^{(4)} = a^{(4)} - y$$