etcd Architecture Explained

Every Kubernetes cluster has one component that is more important than all the others combined.

That component is etcd.

If the API Server is the brain of Kubernetes, etcd is its memory.

Everything Kubernetes knows — every Pod, Node, Deployment, Service, Secret, RBAC rule, lease — lives inside etcd.

flowchart LR

User["kubectl"]

-->APIServer["API Server"]

APIServer

-->etcd["etcd Cluster"]

etcd

-->APIServer

APIServer

-->Scheduler & Controller & Kubelet

Neither the Scheduler nor the Controller Manager talks directly to etcd. Everything goes through the API Server — which means etcd only has one client to worry about.

---

Why etcd?

Kubernetes needed a storage layer with three properties that are difficult to combine:

Property	Why Kubernetes needs it
Strong consistency	Two controllers must never see conflicting state
High availability	Control plane survives node failures without data loss
Watch API	Controllers react to changes instantly rather than polling

etcd provides all three. A standard relational database gives you consistency but not the watch API. A cache gives you watches but not durability. etcd was purpose-built for the control-plane use case.

---

High Availability Architecture

A production etcd cluster runs an odd number of nodes — always 3, 5, or 7.

flowchart LR

APIServer["API Server"]

-->Leader

Leader

-->Follower1

Leader

-->Follower2

Follower1

<-->Follower2

Odd numbers are required because etcd uses majority voting. An even number risks a split-brain: two equally sized groups, each believing the other is down, both trying to elect a leader.

Cluster size	Nodes needed for majority	Tolerated failures
3	2	1
5	3	2
7	4	3

Most production clusters use 3 nodes. 5 nodes are used when the control plane is spread across 3 availability zones and you need to tolerate a full zone failure.

---

Raft Consensus Algorithm

etcd uses Raft to keep all nodes in sync.

At any moment there is exactly one Leader. All writes go to the Leader. Followers only replicate.

Write Path

sequenceDiagram

participant Client
participant API as API Server
participant Leader
participant F1 as Follower 1
participant F2 as Follower 2

Client->>API: Create Pod
API->>Leader: Write entry
Leader->>F1: AppendEntries (replicate log)
Leader->>F2: AppendEntries (replicate log)
F1-->>Leader: ACK
F2-->>Leader: ACK
Leader-->>API: Commit (majority received)
API-->>Client: 201 Created

A write is committed only after a majority acknowledges it. With 3 nodes, 2 ACKs are enough — so etcd can absorb one node failure mid-write without data loss.

Read Path

The API Server performs linearizable reads from the Leader by default — it sends a read with the current revision and the Leader confirms it has not been superseded. This guarantees Kubernetes controllers always observe the latest committed state.

Leader Election

When the Leader disappears, followers hold an election:

flowchart TD

Leader["Leader — crashes"]

-->Timeout["Followers detect heartbeat timeout\n(default: 1s election timeout)"]

Timeout

-->Vote["Followers request votes"]

Vote

-->NewLeader["First to get majority → new Leader"]

NewLeader

-->Resume["Cluster resumes\n(typically <2s)"]

During the election window, writes are blocked — the API Server queues them. Reads from the Watch Cache still work because the cache is in-memory.

---

Raft Log

Every committed write becomes an entry in the Raft log — an append-only, ordered sequence of operations.

Index   Term   Operation
1       1      Create Namespace "default"
2       1      Create Pod "nginx"
3       2      Update Deployment "web" replicas=3
4       2      Delete Pod "nginx"
5       3      Create Service "web-svc"

Every node stores the same log. Followers that fall behind catch up by replaying entries they missed. This is how a restarted node rejoins the cluster without a full resync.

---

MVCC — Multi-Version Concurrency Control

etcd never overwrites data in place. Every update creates a new revision.

Revision 10:  pod/nginx  →  phase=Pending
Revision 11:  pod/nginx  →  phase=Running
Revision 12:  pod/nginx  →  phase=Succeeded

Revisions are monotonically increasing across the entire cluster — not just per object. Every write to any key advances the global revision counter.

The resourceVersion Connection

This is the same number exposed as resourceVersion in every Kubernetes object:

metadata:
  name: nginx
  resourceVersion: "12"   # ← this is the etcd revision at last write

When two controllers try to update the same object simultaneously:

• Controller A reads revision 12, sends PUT with resourceVersion: "12" — succeeds, new revision is 13
• Controller B also read revision 12, sends PUT with resourceVersion: "12" — etcd rejects it (revision 12 is stale)
• Controller B receives 409 Conflict, re-fetches at revision 13, retries

This is optimistic concurrency without any locking. etcd's MVCC is what makes it safe.

MVCC and Watches

MVCC also powers the Watch API. A client can watch from any past revision:

WATCH /pods  startRevision=11

etcd replays all events since revision 11 and then streams new ones. If a controller disconnects and reconnects, it resumes the watch from its last-seen revision — no events are missed.

---

Snapshots and Compaction

Snapshots

The Raft log grows unboundedly. Replaying millions of entries after a restart would take minutes.

etcd solves this with periodic snapshots — a point-in-time serialization of the entire key-value store.

flowchart LR

RaftLog["Raft Log\nentries 1 – 5,000,000"]

-->Snapshot["Snapshot\n(state at entry 5,000,000)"]

Snapshot

-->NewLog["New log\nentries 5,000,001+"]

On restart, etcd loads the latest snapshot and replays only the log entries after it. Recovery time drops from minutes to seconds.

etcd triggers a snapshot automatically when the log exceeds --snapshot-count entries (default: 100,000).

Log Compaction

After a snapshot, old log entries are no longer needed. Compaction discards them and reclaims disk space.

Before compaction:  entries 1 – 5,000,000  (large)
After compaction:   entries 4,900,001 – 5,000,000  (small)

The entries before the snapshot can be safely discarded because the snapshot already captures that state.

Database Defragmentation

MVCC keeps old revisions alive until compaction removes them, but compaction leaves holes in the backend database file — it does not shrink the file on disk.

etcdctl defrag rewrites the backend file, reclaiming the holes:

etcdctl defrag --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Run defrag on followers first, then the leader. Each node is briefly unavailable during defrag, so running one at a time prevents quorum loss.

---

The Watch API

Controllers don't poll etcd. They subscribe to a prefix and receive events.

sequenceDiagram

participant Controller
participant APIServer as API Server
participant etcd

Controller->>APIServer: WATCH /pods (startRevision=500)
APIServer->>etcd: WATCH /pods (startRevision=500)

Note over etcd: Pod nginx created (revision 501)
etcd-->>APIServer: ADDED nginx
APIServer-->>Controller: ADDED nginx

Note over etcd: Pod nginx updated (revision 502)
etcd-->>APIServer: MODIFIED nginx
APIServer-->>Controller: MODIFIED nginx

Event types:

Event	Meaning
ADDED	Key created
MODIFIED	Key updated (new revision)
DELETED	Key removed

The controller reconnects with its last-seen resourceVersion if the watch stream drops. etcd can serve events from its compacted history only up to the oldest retained revision — if a controller is too far behind it receives a 410 Gone and must re-list from scratch.

---

Quorum Loss — What Actually Happens

Losing quorum (majority of nodes) does not crash Kubernetes. It freezes write operations while keeping reads alive from the API Server's Watch Cache.

flowchart TD

TwoOfThree["2 of 3 etcd nodes fail"]

-->NoQuorum["etcd has no quorum\nwrites blocked"]

NoQuorum

-->APIReads["API Server serves reads\nfrom Watch Cache (stale)"]

NoQuorum

-->APIWrites["API Server rejects writes\n503 Service Unavailable"]

APIReads

-->ControlPlane["Controllers keep reconciling\nagainst cached state\n(no new state changes take effect)"]

Existing Pods keep running — kubelet is independent and does not need etcd to run containers. New Pod creates, scaling operations, and config changes are blocked until quorum is restored.

This is why etcd HA and backup are treated as cluster-critical — losing the etcd majority stops the entire control plane from accepting changes.

---

etcdctl — Operational Commands

etcdctl is the CLI for managing etcd. Always set the v3 API:

export ETCDCTL_API=3
ETCD_FLAGS="--endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key"

Check cluster health:

etcdctl $ETCD_FLAGS endpoint health
# https://127.0.0.1:2379 is healthy: successfully committed proposal

etcdctl $ETCD_FLAGS endpoint status --write-out=table
# ENDPOINT           ID    VERSION  DB SIZE  IS LEADER  RAFT TERM  RAFT INDEX
# 127.0.0.1:2379  abc123  3.5.9    42 MB    true       5          1048576

Take a snapshot (backup):

etcdctl $ETCD_FLAGS snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db

etcdctl snapshot status /backup/etcd-snapshot-20260707.db --write-out=table
# HASH     REVISION  TOTAL KEYS  TOTAL SIZE
# a1b2c3d4  1048576   12345       42 MB

Restore from snapshot:

etcdctl snapshot restore /backup/etcd-snapshot-20260707.db \
  --name etcd-0 \
  --initial-cluster etcd-0=https://10.0.0.1:2380 \
  --initial-advertise-peer-urls https://10.0.0.1:2380 \
  --data-dir /var/lib/etcd-restore

Read a Kubernetes object directly from etcd:

etcdctl $ETCD_FLAGS get /registry/pods/default/nginx --print-value-only \
  | protoc --decode_raw   # objects are stored as protobuf, not JSON

---

Performance Metrics That Matter

etcd performance determines control plane latency. Key metrics exposed via Prometheus:

Metric	What it measures	Alert threshold
etcd_disk_wal_fsync_duration_seconds	Time to flush WAL to disk	p99 > 10 ms
etcd_disk_backend_commit_duration_seconds	Time to commit a batch to BoltDB	p99 > 25 ms
etcd_server_leader_changes_seen_total	Leader elections since start	> 3 in 1 hour
etcd_server_proposals_failed_total	Failed Raft proposals	Any increase
etcd_mvcc_db_total_size_in_bytes	Raw DB file size	Alert at 6 GB (hard limit 8 GB)
etcd_mvcc_db_total_size_in_use_in_bytes	Actual data in use	Difference from above = defrag needed

The WAL fsync duration is the most sensitive indicator of disk health. SSD-backed etcd should stay under 1 ms p99. Network-attached storage (NFS, EBS gp2) regularly exceeds 10 ms — the main reason cloud providers run etcd on locally-attached NVMe.

---

Key Takeaways

Concept	Purpose
Raft	Leader-based consensus — all writes replicated to majority before commit
Leader Election	Automatic failover in < 2 s when leader crashes
Raft Log	Ordered, append-only record of every change — enables follower catch-up
MVCC	Every write creates a new revision — enables watches and optimistic concurrency
resourceVersion	The etcd revision number exposed on every Kubernetes object
Snapshots	Point-in-time state capture — fast recovery without log replay
Compaction	Removes old MVCC revisions to reclaim storage
Defrag	Shrinks the on-disk BoltDB file after compaction
Watch API	Event stream from a revision — controllers subscribe rather than poll
Quorum loss	Writes blocked, reads from cache still work, Pods keep running

etcd is the only stateful component in the Kubernetes control plane. Everything else can be restarted from scratch — the API Server, Scheduler, and Controller Manager are all stateless. etcd cannot. That is why backup, HA, and disk latency are treated as cluster-critical concerns.