Neural Networks Overview

The Feature Explosion Problem

Why We need Neural Networks?

Suppose we have $x_1, x_2, \dots, x_n$ as input features and we want to compute a hypothesis $h_\theta(x)$.

We can

$g(\theta_0+ \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_n x_n)$

For quadratic features

$g(\theta_0+ \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_n x_n + \theta_{n+1} x_1^2 + \dots)$

• Quadratic terms grow rougly as $n^2/2$

so we will end up with 5000 additional features if we have 100 features.

For cubic features

$g(\theta_0+ \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_n x_n + \theta_{n+1} x_1^2 + \dots + \theta_{n+k} x_1^3 + \dots)$

• Cubic terms grows as $O(n^3)$

• So we will end up with 166,000 additional features if we have 100 features.

As the features become more complex, the number of parameters $\theta$ grows rapidly.

It becomes

• computationally expensive to compute the hypothesis with many features.
• Memory Heavy to store all the parameters.
• Prone to overfitting due to the large number of parameters.

In this case, we can use a neural network to compute the hypothesis more efficiently.

Practical Example: Image Recognition

Suppose we have a 100 × 100 pixel image as input.

• Each pixel is a feature, so we have 10,000 features.
• For RGB images, we have 3 color channels, so we have 30,000 features.

If we want to compute a hypothesis with quadratic features, we would have on the order of 450 million features, which is computationally infeasible.

Conclusion

Polynomial logistic regression: Works for small $𝑛$

• Explodes combinatorially for large $𝑛$
• Computationally impossible for large feature sets (like images)
• We need a non-linear model that can capture complex relationships without explicitly generating all polynomial features.

---

Neural Networks as a Solution

Neural networks can compute complex hypotheses without explicitly generating all polynomial features.

Neurons as Computational Units

At a simple level, neurons are computational units.

They:

• dendrites(head): Take inputs $x_1, x_2, \dots, x_n$
• Process them: apply weights and activation function
• axon(tail): Produce an output $h_\theta(x)$

Artificial Neurons

In artificial neural networks, we model neurons as mathematical functions.

In our machine learning model:

Inputs are features: $x_1, x_2, \dots, x_n$

$$\begin{bmatrix} x_0 \\ x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}$$

• $x_0$ is the bias unit, and it is always equal to 1

Neuron Does complex transformations on the input features:

$$\theta = \begin{bmatrix} \theta_0 \\ \theta_1 \\ \theta_2 \\ \vdots \\ \theta_n \end{bmatrix}$$

Output is the hypothesis $h_\theta(x)$

where $h_\theta(x) = g(\theta^T x)$

And it can be rewritten as: $z = \theta^T x$

so the hypothesis can be expressed as: $h_\theta(x) = g(z)$

$g(z)$ is the activation function that introduces non-linearity into the model.

Activation Function

$g(z)$ is the sigmoid activation function. for Hypothesis function:

Neural networks use Logistic (Sigmoid) as logistic regression:

$h_\theta(x) = g(z)$

$$g(z) = \frac{1}{1 + e^{-z}}$$

For a single unit:

$$h_\theta(x) = \frac{1}{1 + e^{-\theta^T x}}$$

This function is called the sigmoid activation function.

In neural networks:

• Parameters $\theta$ are called weights
• Outputs of neurons are called activations

---

Neural Network Structure

A Artificial Neural Network is a computational graph with layers of artificial neurons.

A simple network looks like:

$$\begin{bmatrix} x_0 \\ x_1 \\ x_2 \end{bmatrix} \rightarrow \text{Neuron} \rightarrow h_\theta(x)$$

Complex networks have multiple layers:

$$\text{Input} \rightarrow \text{Hidden} \rightarrow \text{Output}$$ $$\begin{bmatrix} x_0 \\ x_1 \\ x_2 \\ x_3 \end{bmatrix} \rightarrow \begin{bmatrix} a^{(2)}_1 \\ a^{(2)}_2 \\ a^{(2)}_3 \end{bmatrix} \rightarrow h_\theta(x)$$

graph LR
  Input --> Hidden-Layer
  Hidden-Layer--> Output

• Layer 1: Input layer: takes features $x_1, x_2, \dots, x_n$ and bias unit $x_0$
• Layer 2: Hidden layer: computes intermediate activations $a^{(2)}_1, a^{(2)}_2, \dots$
• Layer n: Output layer: gives us the final hypothesis $h_\theta(x)$

Where:

$a^{(j)}_i$ = activation of unit $i$ in layer $j$

• $a_1^{(2)}$: first neuron in hidden layer

  $\Theta^{(j)}$ = weight matrix mapping layer $j$ to layer $j+1$

Each Input Layer is densely connected to each Activation Function of next Layer:

Top Down

graph 

subgraph Input Layer
x1(((x1)))
x2(((x2)))
x3(((x3)))
end

subgraph Hidden Layers
a1{a1}
a2{a2}
a3{a3}
end

subgraph Output_Layer
y(((hθx)))
end

x1 --> a1
x1 --> a2
x1 --> a3

x2 --> a1
x2 --> a2
x2 --> a3

x3 --> a1
x3 --> a2
x3 --> a3

a1 --> y
a2 --> y
a3 --> y

IN Left to Right

graph LR

subgraph Input Layer
    x1(((x1)))
    x2(((x2)))
    x3(((x3)))
end

subgraph Hidden Layer
    a1{a1}
    a2{a2}
    a3{a3}
end

subgraph Output Layer
    y(((hθx)))
end

x1 --> a1
x1 --> a2
x1 --> a3

x2 --> a1
x2 --> a2
x2 --> a3

x3 --> a1
x3 --> a2
x3 --> a3

a1 --> y
a2 --> y
a3 --> y

Simplified:

graph LR

x1(((x1))) --> a1{a1}
x1 --> a2{a2}
x1 --> a3{a3}

x2(((x2))) --> a1
x2 --> a2
x2 --> a3

x3(((x3))) --> a1
x3 --> a2
x3 --> a3

a1 --> y(((hθx)))
a2 --> y
a3 --> y

Advance 4 Layer Neural Network:

graph LR

%% Input Layer
subgraph Input Layer
x1(((x1)))
x2(((x2)))
x3(((x3)))
end

%% Hidden Layer 1
subgraph Hidden Layer 1
a1{a1}
a2{a2}
a3{a3}
end

%% Hidden Layer 2
subgraph Hidden Layer 2
b1{b1}
b2{b2}
b3{b3}
end

%% Output Layer
subgraph Output Layer
y(((hθx)))
end

%% Connections: Input → Hidden 1
x1 --> a1
x1 --> a2
x1 --> a3

x2 --> a1
x2 --> a2
x2 --> a3

x3 --> a1
x3 --> a2
x3 --> a3

%% Connections: Hidden 1 → Hidden 2
a1 --> b1
a1 --> b2
a1 --> b3

a2 --> b1
a2 --> b2
a2 --> b3

a3 --> b1
a3 --> b2
a3 --> b3

%% Connections: Hidden 2 → Output
b1 --> y
b2 --> y
b3 --> y

---

Neural Network Hypothesis

Neural networks are simply multiple logistic regression units stacked together.

Each layer:

1. Takes activations from previous layer
2. Multiplies by weight matrix
3. Applies sigmoid activation
4. Passes result forward

Computing Hidden Layer Activations

Each hidden unit is computed as:

$$a^{(2)}_1 = g(\Theta^{(1)}_{10}x_0 + \Theta^{(1)}_{11}x_1 + \Theta^{(1)}_{12}x_2 + \Theta^{(1)}_{13}x_3)$$ $$a^{(2)}_2 = g(\Theta^{(1)}_{20}x_0 + \Theta^{(1)}_{21}x_1 + \Theta^{(1)}_{22}x_2 + \Theta^{(1)}_{23}x_3)$$ $$a^{(2)}_3 = g(\Theta^{(1)}_{30}x_0 + \Theta^{(1)}_{31}x_1 + \Theta^{(1)}_{32}x_2 + \Theta^{(1)}_{33}x_3)$$

This means:

• We use a 3 × 4 matrix of weights
• Each row corresponds to one hidden unit
• Each row multiplies all input features (including bias)

---

Output Layer

The final hypothesis is:

$$h_\Theta(x) = a^{(3)}_1$$ $$a^{(3)}_1 = g(\Theta^{(2)}_{10}a^{(2)}_0 + \Theta^{(2)}_{11}a^{(2)}_1+ \Theta^{(2)}_{12}a^{(2)}_2+ \Theta^{(2)}_{13}a^{(2)}_3)$$

So:

• Hidden layer outputs become inputs to the next layer
• Another weight matrix $\Theta^{(2)}$ is applied
• Then the sigmoid function is applied again

---

Weight Matrix Dimensions

If:

• Layer $j$ has $s_j$ units
• Layer $j+1$ has $s_{j+1}$ units

Then:

$$\Theta^{(j)} \in \mathbb{R}^{\, s_{j+1} \times (s_j + 1)}$$

Why the +1?

Because of the bias unit.

Important detail:

• Input side includes bias
• Output side does NOT include bias

---