Multiclass Classification with Neural Networks

Multiclass Classification with Neural Networks

Extending Binary Classification

In binary classification, our hypothesis outputs a single value:

$$h_\Theta(x) \in [0,1]$$

For multiclass classification, instead of returning a single value,
our hypothesis returns a vector of probabilities.

Simplified Cost

This is the binary cost for a single output unit — useful for building intuition, but it is not the multi-class cost itself (there's no sum over classes k yet). If we ignore multiclass and regularization, the cost for training example $t$ is:

$$\text{cost}(t) = y^{(t)} \log(h_\Theta(x^{(t)})) + (1 - y^{(t)}) \log(1 - h_\Theta(x^{(t)}))$$

The actual multi-class cost sums this expression over every output unit k (and adds regularization) — see 6-Cost-Function for the full formula.

---

Example: Four-Class Classification

Suppose we want to classify an image into one of four categories:

1. 🚗 Car
2. 🏃 Pedestrian
3. 🚚 Truck
4. 🛵 Motorcycle

Instead of one output unit, we use four output units.

Network Structure

$$\text{Input Image} \rightarrow \text{Hidden Layers} \rightarrow \text{4 Output Units}$$

Each output unit corresponds to one class.

graph LR

%% Styling
    classDef input fill:#1e293b,stroke:#38bdf8,color:#ffffff,stroke-width:2px;
    classDef output fill:#111827,stroke:#f59e0b,color:#ffffff,stroke-width:2px;


%% ===== Input Layer =====
    subgraph "Input Layer"
        x0(((x0=1)))
        x1(((x1)))
        x2(((x2)))
        x3(((x3)))
    end

    %% ===== Hidden Layer =====
    subgraph "Hidden Layer"
        a0(((a0=1)))
        a1{a1}
        a2{a2}
        a3{a3}
    end

    %% ===== Output Layer =====
    subgraph "Output Layer (4 Classes)"
        y1(((hθx1 = 🚗)))
        y2(((hθx2 = 🏃‍)))
        y3(((hθx3 = 🚚)))
        y4(((hθx4 = 🛵)))
    end


    %% Input → Hidden
    x0 --> a1
    x0 --> a2
    x0 --> a3
    x1 --> a1
    x1 --> a2
    x1 --> a3
    x2 --> a1
    x2 --> a2
    x2 --> a3
    x3 --> a1
    x3 --> a2
    x3 --> a3

    %% Hidden → Output
    a0 --> y1
    a0 --> y2
    a0 --> y3
    a0 --> y4
    a1 --> y1
    a1 --> y2
    a1 --> y3
    a1 --> y4
    a2 --> y1
    a2 --> y2
    a2 --> y3
    a2 --> y4
    a3 --> y1
    a3 --> y2
    a3 --> y3
    a3 --> y4


%% Assign classes
class x1,x2,x3 input
class y1,y2,y3,y4 output

Output Representation

Our hypothesis now returns:

$$h_\Theta(x) = \begin{bmatrix} h_\Theta(x)_1 \\ h_\Theta(x)_2 \\ h_\Theta(x)_3 \\ h_\Theta(x)_4 \end{bmatrix}$$

Where:

• $h_\Theta(x)_1$ → Probability of Car
• $h_\Theta(x)_2$ → Probability of Pedestrian
• $h_\Theta(x)_3$ → Probability of Truck
• $h_\Theta(x)_4$ → Probability of Motorcycle

Training Labels (One-Hot Encoding)

Each training example has a label vector:

$$y^{(i)} \in \mathbb{R}^4$$

Examples:

Car:

$$\begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \end{bmatrix}$$

Motorcycle:

$$\begin{bmatrix} 0 \\ 0 \\ 0 \\ 1 \end{bmatrix}$$

This is called one-hot encoding.

Example Output

In practice the network rarely outputs exact 0s and 1s — each unit outputs a probability, e.g.:

$$h_\Theta(x) = \begin{bmatrix} 0.02 \\ 0.05 \\ 0.90 \\ 0.03 \end{bmatrix}$$

We pick the predicted class via:

$$\arg\max_k \ h_\Theta(x)_k = 3$$

So the predicted class is the third category (highest probability), even though the output isn't an exact one-hot vector.

If we defined:

1 → Car
2 → Pedestrian
3 → Truck
4 → Motorcycle

Then the model predicts:

$$\text{Truck}$$

Decision Rule

In practice, we select:

$$\text{Prediction} = \arg\max_k h_\Theta(x)_k$$

That is, we choose the class with the largest output value.

---

Key Idea

• Binary classification → 1 output unit
• Multiclass classification → K output units
• Output layer size = number of classes
• Final prediction = index of largest output

Neural networks naturally extend logistic regression to multiple classes by simply increasing the number of output neurons.

---

Modern Relevance

Every LLM generating text is doing exactly this — multi-class classification — on every single token it produces.

Llama 3.1's vocabulary is 128,256 tokens. At each generation step, the model runs a forward pass and produces a 128,256-dimensional output vector. Softmax converts it to a probability distribution. The selected token is the "predicted class."

• Temperature sampling modifies the softmax: dividing logits by temperature T before softmax makes the distribution sharper (T < 1, more confident) or flatter (T > 1, more random). T = 0 is argmax — always pick the highest-probability class.
• Top-k and top-p sampling are post-softmax filters: restrict generation to the top-k most probable tokens, or the smallest set of tokens whose cumulative probability exceeds p. These map to restricting the K-class output to a subset before drawing.
• One-hot encoding as training labels is exactly how LLM pre-training works: the correct next token is a one-hot vector over 128K classes, and cross-entropy loss penalizes the model for assigning low probability to the true next token.

At 70B tokens/second across a DGX cluster, the system runs this 128K-class softmax millions of times per second — hardware-fused into the final linear layer by TensorRT-LLM for maximum throughput.

---

Related Posts

• Logistic Regression II: Multi-Class and One-vs-All — neural networks replace the K separate one-vs-all classifiers with a single network and one-hot output labels
• Cost Function for Neural Networks — the multi-class cost function is the generalized version of what's defined here applied across K output nodes
• Examples and Intuitions II — the binary-output network this generalizes to multiple classes