Neural Networks Overview

The Feature Explosion Problem

Why oo we need Neural Networks?

Suppose we have $x_1, x_2, \dots, x_n$ as input features and we want to compute a hypothesis $h_\theta(x)$.

We can

$g(\theta_0+ \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_n x_n)$

For quadratic features

$g(\theta_0+ \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_n x_n + \theta_{n+1} x_1^2 + \dots)$

• Quadratic terms grow roughly as $n^2/2$

so we will end up with 5000 additional features if we have 100 features.

For cubic features

$g(\theta_0+ \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_n x_n + \theta_{n+1} x_1^2 + \dots + \theta_{n+k} x_1^3 + \dots)$

• Cubic terms grows as $O(n^3)$

• So we will end up with 166,000 additional features if we have 100 features.

As the features become more complex, the number of parameters $\theta$ grows rapidly.

It becomes

• computationally expensive to compute the hypothesis with many features.
• Memory-Heavy to store all the parameters.
• Prone to overfitting due to the large number of parameters.

In this case, we can use a neural network to compute the hypothesis more efficiently.

Practical Example: Image Recognition

Suppose we have a 100 × 100 pixel image as input.

• Each pixel is a feature, so we have 10,000 features.
• For RGB images, we have 3 color channels, so we have 30,000 features.

If we want to compute a hypothesis with quadratic features, we would have on the order of 450 million features, which is computationally infeasible.

Conclusion

Polynomial logistic regression: Works for small $𝑛$

• Explodes combinatorially for large $𝑛$
• Computationally impossible for large feature sets (like images)
• We need a non-linear model that can capture complex relationships without explicitly generating all polynomial features.

---

Neural Networks as a Solution

Neural networks can compute complex hypotheses without explicitly generating all polynomial features.

NN Types and Applications

Comparison of Neural Network Types

Network Type	Best For	Memory	Spatial Awareness
Feedforward	Tabular data	No	No
CNN	Images	No	Yes
RNN	Sequences	Yes	Limited

Evolution of Architectures

Today, many RNN tasks are increasingly replaced by:

• Transformers
• Attention mechanisms

flowchart LR

    A[Feedforward Networks]

    A --> B[CNNs]

    A --> C[RNNs]

    C --> D[LSTM / GRU]

    D --> E[Transformers]

1. 🔀 Standard Feedforward Networks

Information moves only in one direction:

Also called:

• Fully Connected Networks
• Dense Neural Networks
• Multi-Layer Perceptrons (MLP)

$$\text{Input} \rightarrow \text{Hidden Layers} \rightarrow \text{Output}$$

No cycles or memory.

flowchart LR

    A1[Input 1]
    A2[Input 2]
    A3[Input 3]

    A1 --> H1
    A1 --> H2

    A2 --> H1
    A2 --> H2

    A3 --> H1
    A3 --> H2

    H1[Hidden Layer]
    H2[Hidden Layer]

    H1 --> O[Output]
    H2 --> O

Often used for:

• Housing price prediction
• Online advertising
• Credit scoring
• Customer churn prediction
• Fraud detection

Mathematical Representation

For one layer:

$$y = f(Wx + b)$$

Where:

• $x$ = input vector
• $W$ = weights
• $b$ = bias
• $f$ = activation function

2. 🏞️ Convolutional Neural Networks (CNNs)

CNNs are specialized neural networks designed primarily used for image data

• Exploit spatial structure in images

Instead of connecting every neuron to every pixel to CNNs detect patterns using

• convolution filters
• feature maps

flowchart TD

    A[Input Image]

    A --> B[Convolution Layer]

    B --> C[Feature Maps]

    C --> D[Pooling Layer]

    D --> E[Fully Connected Layer]

    E --> F[Prediction]

Mathematical Representation of Convolution

A filter slides across the image.

$$(I * K)(x,y) = \sum_m \sum_n I(m,n)K(x-m,y-n)$$

Where:

• $I$ = image
• $K$ = kernel/filter
• $K$ = kernel/filter

flowchart LR

    A[Image]

    A --> B[Edges]

    B --> C[Textures]

    C --> D[Shapes]

    D --> E[Objects]

Use Case

• Image classification
• Face recognition
• Medical imaging
• Self-driving cars
• Object detection

3. 🔢 Recurrent Neural Networks (RNNs)

RNNs process data step-by-step while remembering previous information.

• Unlike feedforward networks, RNNs have memory.
• They can capture temporal dependencies in data.

Used for sequence data

Examples:

• Audio (time series)
• Language (word-by-word sequence)

RNN Architecture

flowchart LR

    X1[Word 1] --> H1
    H1 --> H2

    X2[Word 2] --> H2
    H2 --> H3

    X3[Word 3] --> H3

    H1[Hidden State]
    H2[Hidden State]
    H3[Hidden State]

Mathematical Representation of RNNs

$$h_t = f(x_t, h_{t-1})$$

Where:

• $x_t$ = current input
• $h_{t-1}$ = previous hidden state
• $h_t$ = updated memory state

Use Cases for RNNs

Audio / Time Series

• Speech recognition
• Sensor data
• Financial forecasting

Language Processing

• Translation
• Chatbots
• Text generation
• Next

4. Custom Neural Networks

Tailored for specific applications Used in complex systems like autonomous driving:

• CNNs for images
• Other components for radar
• Combined into custom architectures

---

Neurons as Computational Units

At a simple level, neurons are computational units.

They:

• dendrites(head): Take inputs $x_1, x_2, \dots, x_n$
• Process them: apply weights and activation function
• axon(tail): Produce an output $h_\theta(x)$

Artificial Neurons Model: Logistic Unit

In artificial neural networks, we model neurons as mathematical Logistic functions.

graph LR

subgraph Input Layer
x0(((x0)))    
x1(((x1)))
x2(((x2)))
x3(((x3)))
end

subgraph Activation Layer
a1{a1}
end

x0-->a1
x1-->a1
x2-->a1
x3-->a1


subgraph Output Layer
y(((hθx)))
end

a1-->y

A simple network looks like:

$$\begin{bmatrix} x_0 \\ x_1 \\ x_2 \\ x_3 \end{bmatrix} \rightarrow \text{Neuron} \rightarrow h_\theta(x)$$

In our machine learning model:

Inputs are features: $x_1, x_2, \dots, x_n$

$$x= \begin{bmatrix} x_0 \\ x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}$$

Parameters $\theta$ are called weights

$$\theta = \begin{bmatrix} \theta_0 \\ \theta_1 \\ \theta_2 \\ \vdots \\ \theta_n \end{bmatrix}$$

Bias unit

$x_0$ is the bias unit/ bias Neuron and it is always equal to 1

• For simplicity we dont draw $x_0$

Output

Outputs of neurons are called activations Weights which is the hypothesis $h_\theta(x)$

where $h_\theta(x) = g(\theta^T x)$

And we can rewrite: $z = \theta^T x$

so the hypothesis can be expressed as: $h_\theta(x) = g(z)$

$g(z)$ is the activation function that introduces non-linearity into the model.

Activation Function $g(\cdot)$

$g(z)$ is the activation function. for Hypothesis function:

• Example: ReLU, sigmoid, tanh, etc.

Neural networks using sigmoid activation function for logistic regression:

$h_\Theta(x) = g(z)$

Where

$$g(z) = \frac{1}{1 + e^{-z}}$$

Where $z = \theta^T x$

ReLU vs Sigmoid Activation Function

Sigmoid

Input:  -5    0    5
Output: 0.01 0.5 0.99

Outputs behave like probabilities.

Sigmoid squashes outputs between:

0 and 1

Its derivative becomes very small for large positive or negative values.

Vanishing gradient

Deep neural networks may contain:

• dozens
• hundreds
• thousands

of layers.

During backpropagation:

• gradients are multiplied repeatedly across layers.

If gradients are small:

0.1 × 0.1 × 0.1 × 0.1 ...

they rapidly approach zero.

Example

Suppose gradient values are:

0.2 × 0.2 × 0.2 × 0.2 × 0.2

Result:

0.00032

Very tiny gradients mean:

• almost no weight updates
• slow or stalled learning

Exploding vs Vanishing Gradients

Problem	What Happens
Vanishing gradients	Gradients become too small
Exploding gradients	Gradients become too large

Rectified linear unit ReLU

ReLU activates only positive values. Negative values become zero.

Input:  -3  -1   0   2   5
Output:  0   0   0   2   5

ReLU helps because:

• gradients remain stronger
• training becomes faster
• deep networks scale better

Enables: This enables:

• deeper networks
• faster convergence
• stable learning

Which was not possible with sigmoid before

Feature	ReLU (Rectified Linear Unit)	Sigmoid
How it works	If the signal is positive, it passes it forward; if negative, it outputs zero	Converts output into a probability between 0 and 1
Example (Cat model)	Strong whisker detection → output 3.5 (feature exists strongly)	Final output → 0.92 → “92% chance this is a cat”
Output Range	[0, ∞)	[0, 1]
Best suited for	Hidden layers; very fast and helps deep models learn efficiently	Output layer for yes/no classification problems
Performance	Computationally efficient	Comparatively slower
Formula	f(x) = max(0, x)	f(x) = 1 / (1 + e^(-x))
Optimized for	Avoiding vanishing gradients	Predicting probabilities

Layer Normalization

Layer Normalization is a technique used in neural networks to normalize the activations of a layer for each individual training example.

it transforms activations so they have:

• mean approximately $0$
• variance approximately $1$

Most values often fall within a few standard deviations of zero, which is why ranges like $-3$ to $+3$ are commonly observed.

It helps:

• stabilize training
• speed up convergence
• reduce sensitivity to initialization
• prevent activations from becoming too large or too small

Layer Normalization computes the mean and variance across the features of a single sample.

The normalization formula is:

$$\hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}}$$

Where:

• $x_i$ = input activation
• $\mu$ = mean of activations in the layer
• $\sigma^2$ = variance
• $\epsilon$ = small constant for numerical stability

After normalization, learnable parameters scale and shift the values:

$$y_i = \gamma \hat{x}_i + \beta$$

Where:

• $\gamma$ = learnable scale parameter
• $\beta$ = learnable shift parameter

BatchNorm vs LayerNorm

Technique	Normalization Across
Batch Normalization	Batch dimension
Layer Normalization	Feature dimension within one sample

Layer Normalization works especially well in Transformer models because it does not depend on batch size.

---

Neural Network Structure

An Artificial Neural Network is a computational graph with layers of artificial neurons.

Neural networks are simply multiple logistic regression units stacked together.

Each layer:

1. Takes activations from previous layer
2. Multiplies by weight matrix
3. Applies sigmoid activation
4. Passes result forward

$$\text{Input} \rightarrow \text{Hidden} \rightarrow \text{Output}$$ $$\begin{bmatrix} x_0 \\ x_1 \\ x_2 \\ x_3 \end{bmatrix} \rightarrow \begin{bmatrix} a^{(2)}_1 \\ a^{(2)}_2 \\ a^{(2)}_3 \end{bmatrix} \rightarrow h_\theta(x)$$

graph LR
    Input --> Hidden-Layer
    Hidden-Layer --> Output

1. Layer 1: Input layer

Takes features as Input

• $x_1, x_2, \dots, x_n$
• $x_0$ = 1 is bias unit that is not drawn

2. Layer 2: Hidden layer:

All Intermediate Layers between Input & Output Layer

• Computes intermediate activations $a^{(2)}_1, a^{(2)}_2, \dots , a^{(2)}_n,$
• $a^{(2)}_0$ =1 is bias unit that is not drawn

$a^{(j)}_i$ = Activation output of $i$ th neuron in layer $j$

• $i$ = Neuron Index inside that layer
• $j$ = layer number

Example:

• $a_1^{(2)}$ = first neuron in Second hidden layer
• $a_3^{(2)}$ = 3rd Neuron in Second hidden layer

Computing Hidden Layer Activations

Activation can be computed as :

$$a^{(2)}_1 = g(\Theta^{(1)}_{10}x_0 + \Theta^{(1)}_{11}x_1 + \Theta^{(1)}_{12}x_2 + \Theta^{(1)}_{13}x_3)$$ $$a^{(2)}_2 = g(\Theta^{(1)}_{20}x_0 + \Theta^{(1)}_{21}x_1 + \Theta^{(1)}_{22}x_2 + \Theta^{(1)}_{23}x_3)$$ $$a^{(2)}_3 = g(\Theta^{(1)}_{30}x_0 + \Theta^{(1)}_{31}x_1 + \Theta^{(1)}_{32}x_2 + \Theta^{(1)}_{33}x_3)$$

Where g(.) is sigmoid

Weight Matrices $\Theta^{(j)}$

Weight Matrix of $jth$ layer

• weight matrix maps layer $j$ to layer $j+1$
• Each $j$ th layer gets its own matrix of weights $\Theta^{(j)}$
• Layers are indexed starting from 1, not 0.

Weight Matrix Dimensions

Outputlayer Neurons x (InputLayer Neurons + 1) Dimensioned Matrix

• Input side includes bias
• Output side does NOT include bias

$\Theta(j)$ would be a Matrix of dimension:

$$s_{(j+1)} X (s_j+1)$$ $$\Theta^{(j)} \in \mathbb{R}^{s_{j+1} \times (s_j + 1)}$$

Where:

• $s_j$ = number of neurons/units in $j$th layer
• $s_{(j+1)}$ units in Output layer $j+1$

Practical Example:

Each Input Layer is densely connected to each Activation Function of next Layer:

• Input layer: 3 units + 1 Bias
• Hidden layer 1: 4 units + 1 Bias
• Output layer: 1 unit

graph LR
    %% ===== Layer 1 =====
    subgraph "Layer 1: Input Layer"
        x0((x0 = 1))
        x1(((x1)))
        x2(((x2)))
        x3(((x3)))
    end

    %% ===== Layer 2 =====
    subgraph "Layer 2: Hidden Layer"
        a0{a0 = 1}
        a1{a1}
        a2{a2}
        a3{a3}
        a4{a4}
    end

    %% ===== Layer 3 =====
    subgraph "Layer 3: Output Layer"
        y(((hθx)))
    end

  
    %% Input → Hidden
    x0 --> a1
    x0 --> a2
    x0 --> a3
    x0 --> a4

    x1 --> a1
    x1 --> a2
    x1 --> a3
    x1 --> a4
  
    x2 --> a1
    x2 --> a2
    x2 --> a3
    x2 --> a4
  
    x3 --> a1
    x3 --> a2
    x3 --> a3
    x3 --> a4


%% Hidden → Output
    a0 --> y
    a1 --> y
    a2 --> y
    a3 --> y
    a4 --> y

Simplified

Showing Bias unit but only 1 connection for reference

graph LR
    x0(((x0))) --> a1{a1}
    x1(((x1))) --> a1{a1}
    x1 --> a2{a2}
    x1 --> a3{a3}
    x1 --> a4{a4}
        
    x2(((x2))) --> a1
    x2 --> a2
    x2 --> a3
    x2 --> a4
  
    x3(((x3))) --> a1
    x3 --> a2
    x3 --> a3
    x3 --> a4

    a0{a0} --> y(((hθx)))
    a1 --> y
    a2 --> y
    a3 --> y
    a4 --> y

Given

• Input layer: $x_0, x_1, x_2, x_3$
• Hidden layer: $a_0, a_1, a_2, a_3, a_4$
• Output layer: $h_\theta(x)$

Where:

• $x_0 = 1$ → bias unit for the input layer
• $a_0 = 1$ → bias unit for the hidden layer

Weight Matrix:

Layer 1 ($S_1$)

Input Layer → Hidden Layer

$$\Theta^{(1)} \in \mathbb{R}^{4 \times 4}$$ $$\Theta^{(1)} = 4 X 4 Matrix$$

=> 4 (a_1, a_2, a_3, a_4) × 4 $(x_0, x_1, x_2, x_3)$

Layer 2 ($S_2$)

Hidden Layer → Output Layer

$$\Theta^{(2)} \in \mathbb{R}^{1 \times 5}$$ $$\Theta^{(2)} = 1 X 5 Matrix$$

=> 1 (output neuron) × 5$(a_0, a_1, a_2, a_3, a_4)$

Activation of Neurons in Layer 2

First Neuron in layer 2

$$a^{(2)}_1 = g(\Theta^{(1)}_{10}x_0 + \Theta^{(1)}_{11}x_1 + \Theta^{(1)}_{12}x_2 + \Theta^{(1)}_{13}x_3)$$

Second Neuron in layer 2

$$a^{(2)}_2 = g(\Theta^{(1)}_{20}x_0 + \Theta^{(1)}_{21}x_1 + \Theta^{(1)}_{22}x_2 + \Theta^{(1)}_{23}x_3)$$

Third Neuron in layer 2

$$a^{(2)}_3 = g(\Theta^{(1)}_{30}x_0 + \Theta^{(1)}_{31}x_1 + \Theta^{(1)}_{32}x_2 + \Theta^{(1)}_{33}x_3)$$

Where

• $g$: sigmoid activation function of collective term

3. Output layer

Gives us the final hypothesis $h_\Theta(x)$

• Hidden layer outputs become inputs to the next layer
• Another weight matrix $\Theta^{(2)}$ is applied
• Then the sigmoid function is applied again

Output Layer Hypothesis

The final hypothesis is the first neuron of 3rd layer

$$h_\Theta(x) = a^{(3)}_1$$

Which is equals to

$$a^{(3)}_1 = g(\Theta^{(2)}_{10}a^{(2)}_0 + \Theta^{(2)}_{11}a^{(2)}_1+ \Theta^{(2)}_{12}a^{(2)}_2+ \Theta^{(2)}_{13}a^{(2)}_3)$$

Where

• $g$: Sigmoid of final term
• $\Theta^{(2)}$ is weight Matrix for final Output Layer

---