Examples and Intuitions I — Neural Networks as Logical Gates

Neural Networks as Logical Gates

A single logistic neuron can simulate logical gates.

By adjusting:

• Bias (threshold)
• Weights (importance of inputs)

we can model:

• AND
• OR
• NOT

Neural networks are powerful because stacking these simple units allows us to represent much more complex functions.

Implementing the AND Operator $(x_1 \land x_2)$

The logical AND operator is true only when:

• $x_1 = 1$
• $x_2 = 1$

Otherwise, it is false.

$x_1$	$x_2$	Result
0	0	0
1	0	0
0	1	0
1	1	1

Network Structure

graph LR

%% Input Layer
    subgraph Input Layer
        x0(((x0)))
        x1(((x1)))
        x2(((x2)))
    end

%% Hidden Layer 1
    subgraph Hidden Layer 1
        a1{a1}
    end

%% Output Layer
    subgraph Output Layer
        y(((hθx)))
    end

%% Connections: Input → Hidden 1
    x0 --> a1
    x1 --> a1
    x2 --> a1
    
%% Connections: Hidden 2 → Output
    a1 --> y
 

Our small neural network looks like:

$$\begin{bmatrix} x_0 \\ x_1 \\ x_2 \end{bmatrix} \rightarrow g(z^{(2)}) \rightarrow h_\Theta(x)$$

Where: $x_0 = 1$ is the bias unit

Choosing the Weights

Consider weight matrix:

$$\Theta^{(1)} = \begin{bmatrix} -30 & 20 & 20 \end{bmatrix}$$

The hypothesis becomes:

$$h_\Theta(x) = g(-30 + 20x_1 + 20x_2)$$

Evaluating All Input Combinations

$x_1$	$x_2$	Expected	$h_\theta(x)$
0	0	0	$g(-30) \approx 0$
1	0	0	$g(-10) \approx 0$
0	1	0	$g(-10) \approx 0$
1	1	1	$g(10) \approx 1$

Conclusion

With this choice of weights:

$$\Theta^{(1)} = \begin{bmatrix} -30 & 20 & 20 \end{bmatrix}$$

the neural network behaves exactly like an AND gate.

---

Implementing the OR Operator $(x_1 \lor x_2)$

The logical OR operator is true when:

• $x_1 = 1$, or
• $x_2 = 1$, or both

$x_1$	$x_2$	Result
0	0	0
1	0	1
0	1	1
1	1	1

We can implement OR using a different set of weights:

$$\Theta^{(1)} = \begin{bmatrix} -10 & 20 & 20 \end{bmatrix}$$

The hypothesis becomes:

$$h_\Theta(x) = g(-10 + 20x_1 + 20x_2)$$

Evaluating All Input Combinations

$x_1$	$x_2$	Expected	$h_\theta(x)$
0	0	0	$g(-10) \approx 0$
1	0	0	$g(10) \approx 1$
0	1	0	$g(10) \approx 1$
1	1	1	$g(30) \approx 1$

Conclusion

With this choice of weights:

$$\Theta^{(1)} = \begin{bmatrix} -10 & 20 & 20 \end{bmatrix}$$

the same neural network behaves exactly like an OR gate.

---

Implementing Not Gate ($\neg x_1$ )

graph LR

%% Input Layer
    subgraph Input Layer
        x0(((x0)))
        x1(((x1)))
    end

%% Hidden Layer 1
    subgraph Hidden Layer 1
        a1{a1}
    end

%% Output Layer
    subgraph Output Layer
        y(((hθx)))
    end

%% Connections: Input → Hidden 1
    x0 --> a1
    x1 --> a1
    
%% Connections: Hidden 2 → Output
    a1 --> y
 

The logical NOT operator is true when:

• $x_1 = 0$

and vice versa

$x_1$	Result
0	1
1	0

We can implement NOT using weights:

$$\Theta^{(1)} = \begin{bmatrix} 10 & -20 \end{bmatrix}$$

The hypothesis becomes:

$$h_\Theta(x) = g(10 - 20x_1)$$

$x_1$	Expected	$h_\theta(x)$
0	1	$g(10) \approx 1$
1	0	$g(-10) \approx 0$

---

Summary

We can use weight to simulate Logic gates with Neural networks

AND

$$\Theta^{(1)} = \begin{bmatrix} -30 & 20 & 20 \end{bmatrix}$$

OR

$$\Theta^{(1)} = \begin{bmatrix} -10 & 20 & 20 \end{bmatrix}$$

NOT

$$\Theta^{(1)} = \begin{bmatrix} 10 & -20 \end{bmatrix}$$

NOR = NOT OR

$$\Theta^{(1)} = \begin{bmatrix} 10 & -20 & -20 \end{bmatrix}$$

---

Modern Relevance

The AND and OR gates here are the simplest possible proof of the universal approximation theorem: a network with enough neurons can represent any Boolean function, and therefore any computable function.

In practice this matters because:

• Attention heads as learned logic gates. Research has found that individual attention heads in transformers implement recognizable patterns — some heads track subject-verb agreement, others copy previous tokens, others implement "previous token" lookup. These are more complex than AND/OR but structurally the same idea: a set of weights that fires on specific input patterns.
• Gating mechanisms. LSTM gates (forget, input, output) and GRU gates are literally sigmoid units computing something close to the soft-AND shown here. The "hard" threshold at 0.5 becomes a differentiable soft gate during training.
• Mixture of Experts (MoE) routing. GPT-4 and Mixtral use a gating network to route each token to one of N expert FFN layers. The router is a small neural network making a soft-argmax decision — a direct descendant of the decision boundary logic demonstrated here.

The weights -30, 20, 20 for AND feel arbitrary, but they encode a precise geometric fact: the decision boundary is a hyperplane that cuts between (1,1) and all other inputs. Modern weight matrices encode millions of such hyperplanes simultaneously.

---

Related Posts

• Vectorized Neural Networks Model Representation — the matrix form these logic-gate examples are built with
• Examples and Intuitions II — extends the logic-gate intuition to a network needing a hidden layer