💰 Cost Function for Neural Networks

Notation

graph LR

%% Input Layer
    subgraph Input Layer
        x1(((x1)))
        x2(((x2)))
        x3(((x3)))
    end

%% Hidden Layer 1
    subgraph Hidden Layer 1
        a1{a1}
        a2{a2}
        a3{a3}
    end

%% Hidden Layer 2
    subgraph Hidden Layer 2
        b1{b1}
        b2{b2}
        b3{b3}
    end

%% Output Layer
    subgraph Output Layer
        y(((hθx)))
    end

%% Connections: Input → Hidden 1
    x1 --> a1
    x1 --> a2
    x1 --> a3
    x2 --> a1
    x2 --> a2
    x2 --> a3
    x3 --> a1
    x3 --> a2
    x3 --> a3
%% Connections: Hidden 1 → Hidden 2
    a1 --> b1
    a1 --> b2
    a1 --> b3
    a2 --> b1
    a2 --> b2
    a2 --> b3
    a3 --> b1
    a3 --> b2
    a3 --> b3
%% Connections: Hidden 2 → Output
    b1 --> y
    b2 --> y
    b3 --> y

Let:

• $m$ = number of training examples: ${(x_1, y_1),(x_2, y_2), \dots, (x_m, y_m) }$
• $K$ = number of output units (classes) eg
  • Binary Classification: K = 1
  • Multi-class Classification: K = 4
• $L$ = total number of layers in the network eg L = 4
• $S_l$ = number of units (excluding bias unit) in layer $l$
  • $S_1= 3$, $S_2=3$, $S_3=3$, $S_4=1$

Since neural networks can have multiple output nodes, we denote $k^{th}$ output as:

$$(h_\Theta(x))_k$$

Neural Network Cost Function

Logistic Regression Cost Function

The regularized logistic regression cost is:

$$J(\theta) =- \frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)} \log(h_\theta(x^{(i)}))+ (1 - y^{(i)}) \log(1 - h_\theta(x^{(i)})) \right] + \frac{\lambda}{2m} \sum_{j=1}^{n} \theta_j^2$$

For a neural network with $K$ output units, the cost becomes:

$$J(\Theta) =- \frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K} \left[ y_k^{(i)} \log((h_\Theta(x^{(i)}))_k) + (1 - y_k^{(i)}) \log(1 - (h_\Theta(x^{(i)}))_k) \right] + \frac{\lambda}{2m} \sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}} (\Theta^{(l)}_{j,i})^2$$

Intuition

Neural network cost function =

• Logistic regression loss applied to every output unit
• Plus regularization over all weights in the network

This is simply a natural extension of logistic regression to multiple layers and multiple outputs.

In short:

$$\text{Neural Network Cost} = \text{Multiclass Logistic Loss}+ \text{Regularization}$$

Double Sum

The double sum adds up logistic regression losses across all output units.

$$\sum_{i=1}^{m} \sum_{k=1}^{K}$$

• The outer sum loops over training examples ($i$)
• The inner sum over all output units ($k$)
• We compute a logistic regression loss for each output node.
• Then we add them together.

This is simply the total loss across all output neurons.

So this is essentially the sum of $K$ logistic regression costs per example.

Triple Sum (Regularization)

The triple sum adds up squared weights across the entire network.

The term

$$\sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}} (\Theta^{(l)}_{j,i})^2$$

means:

• Loop over all layers
• Loops over all units in layer $l$.
• Squares every weight in every $\Theta$ matrix.
• Add them all together

Important:

• Bias weights are not regularized.
• The index $i$ here does not refer to training examples.
• This term regularizes all parameters in the entire network.

---

Modern Relevance

The cost function here — cross-entropy over all output classes plus L2 regularization — is the same function used to pre-train every modern LLM, scaled from a few training examples to trillions of tokens.

Pre-training loss: GPT-4 pre-training minimizes cross-entropy over ~13 trillion tokens of text. The "double sum over training examples and output classes" becomes: sum over all tokens in the dataset × sum over all 128K vocabulary entries (though only the true next-token contributes a non-zero gradient per step).

L2 regularization → weight decay: The triple-sum regularization term (λ/2m) Σ θ² is implemented as weight decay in modern optimizers. AdamW (Adam with decoupled weight decay) applies θ ← θ(1 - λ) after each gradient step — mathematically equivalent, but numerically stable with adaptive learning rates.

Why cross-entropy, not MSE for classification: Cross-entropy has steep gradients when the model is confidently wrong (assigns near-zero probability to the correct class), which MSE doesn't. This makes gradient descent much faster at correcting catastrophic mistakes. At LLM scale, this difference in learning signal per step × trillions of steps compounds significantly.

Mixed precision in production: On H100s, the forward pass (computing this cost) runs in BF16 for throughput. Gradient accumulation runs in FP32 for numerical stability. The cost value itself is a scalar that fits in any precision; the challenge is maintaining accurate gradients through 80 layers of BF16 matrix multiplies.

---

Related Posts

• Logistic Regression: Sigmoid, Decision Boundary, and Classification — the neural network cost function is a direct extension of the logistic regression cross-entropy cost; this post establishes the single-output version
• Regularization: Lasso, Ridge, and the Regularization Parameter — the triple-sum regularization term in the NN cost function applies L2 regularization identically to what's described here
• Backpropagation Algorithm — backpropagation minimizes this cost function; the delta terms are partial derivatives of J with respect to each activation