Image Internals: From OCI Layers to a Running Container

Container Image Internals 📄

Container image is a read-only, layered filesystem with metadata used to create containers.

When you run kubectl apply, a YAML file becomes a running process on a node. That process started from bytes stored in a registry.

Understanding every step between those two points — image pull, layer assembly, snapshot creation, pod sandbox, namespace join — is what separates someone who uses containers from someone who can debug them at any layer of the stack.

---

The Big Picture

Container Anatomy

A container image is a reusable, immutable package, while a container is a running instance of that package.

                    Container Image
    ┌──────────────────────────────────────────────┐
    │ Manifest                                     │
    │ Config (CMD, ENTRYPOINT, ENV, Labels)        │
    ├──────────────────────────────────────────────┤
    │ Layer 4 : Application                        │
    │ Layer 3 : Runtime (Java, Python, Node.js)    │
    │ Layer 2 : Packages                           │
    │ Layer 1 : Base OS (Ubuntu/Alpine)            │
    └──────────────────────────────────────────────┘
                        │
                 OverlayFS Merge
                        │
              Writable Container Layer
                        │
                Running Linux Process

From Image to Container

The runtime combines the image's read-only layers with a writable layer using OverlayFS, then starts the application as a normal Linux process inside isolated namespaces and cgroups.

    Registry (image bytes)
           │
           │ 1. Manifest fetch + layer pull
           ▼
    containerd image store
           │
           │ 2. Snapshot (OverlayFS assembly)
           ▼
    OCI bundle (rootfs + config.json)
           │
           │ 3. Pod sandbox (pause container)
           ▼
    Network namespace + IP assigned
           │
           │ 4. Container joins sandbox
           ▼
    runc creates namespaces, cgroups, pivot_root
           │
           │ 5. exec()
           ▼
    Your process running as PID 1 inside the container

Image vs Container

Image	Container
Read-only	Read-write
Immutable	Mutable
Stored in registry	Running on a host
Blueprint	Executing process
Shared	Isolated instance

---

Step 1: What an Image Actually Is

An OCI image is not a single file. It is three things stored in a content-addressable store:

flowchart TD
Image["Container Image 📄"]

    Image --> Manifest["Manifest 📋"]
    Image --> Config["Image Config 📜"]
    Image --> Layers["Read-only Layers 🗃"️]


    Layers --> Snapshot["Snapshot 💾"]
    Snapshot --> Writable["Writable Layer 📝"]
    Writable --> Container["Container 🐳"]

Component	Purpose
Manifest	Lists layers and configuration
Config	Runtime metadata (CMD, ENTRYPOINT, ENV, etc.)
Layers	Read-only filesystem changes
Digest	Immutable content identifier
Tag	Human-friendly reference

1. The Manifest 📋

The manifest is the table of contents.

The manifest is the index card — it says what layers make up the image and where to find the config.

It tells the runtime

• Which layers exist
• Layer order
• Config file
• Digests

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:a1b2c3...",
    "size": 7023
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d4e5f6...",
      "size": 31379766
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:a7b8c9...",
      "size": 25165824
    }
  ]
}

2. The Config 📜

The config is the runtime metadata

The config stores runtime metadata about:

• what command to run
• Environment variables
• Working directory
• User
• Exposed ports
• Entrypoint

{
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "Env": ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],
    "Cmd": ["nginx", "-g", "daemon off;"],
    "WorkingDir": "/",
    "User": "",
    "ExposedPorts": {"80/tcp": {}}
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:layer1-uncompressed-digest...",
      "sha256:layer2-uncompressed-digest..."
    ]
  },
  "history": [...]
}

3. The Layers 🗃

Each Dockerfile instruction usually creates a new layer.

• Each layer is read-only and can be shared between multiple images.

• Each layer is a gzip-compressed tar archive containing the filesystem diff for that build step. A layer can add files (new path), modify files (whiteout + new version), or delete files (.wh. whiteout marker).

  # A whiteout file tells OverlayFS to hide the file from lower layers
  .wh.etc/passwd        # deletes etc/passwd from the merged view
  .wh..wh..opq          # opaque whiteout — hides entire directory from lower layers
  

Why are images layered?

To reuse common filesystem content, reduce storage, and speed up builds and downloads.

Each Dockerfile instruction usually creates a new layer.

    
    FROM ubuntu:24.04              # Layer 1
    
    RUN apt-get update && \        # Layer 2
        apt-get install -y nginx   # Layer 3
    
    COPY index.html /var/www/html/ # Layer 4
    
    EXPOSE 80                   
    
    CMD ["nginx", "-g", "daemon off;"]
    

How Each Instruction Works

Instruction	Purpose	Creates Image Layer
FROM	Select base image	✅
RUN	Execute commands during build	✅
COPY	Copy local files into image	✅
ADD	Copy files or extract archives	✅
ENV	Set environment variables	❌ (Metadata)
WORKDIR	Set working directory	❌ (Metadata)
EXPOSE	Document listening ports	❌ (Metadata)
CMD	Default startup command	❌ (Metadata)
ENTRYPOINT	Primary executable	❌ (Metadata)

The example above produces an image similar to:

    ┌──────────────────────────────┐
    │ Layer 3                      │
    │ Application Files            │
    │ (COPY index.html)            │
    ├──────────────────────────────┤
    │ Layer 2                      │
    │ Nginx Installation           │
    │ (RUN apt-get install nginx)  │
    ├──────────────────────────────┤
    │ Layer 1                      │
    │ Ubuntu Base Image            │
    │ (FROM ubuntu:24.04)          │
    └──────────────────────────────┘

Writable layer 📝

When a container starts, a writable layer is added on Top of App

What happens when a container modifies a file?

The change is written to the container's writable layer using copy-on-write.

---

Step 2: Content-Addressable Storage #️⃣

Every image has a unique SHA-256 digest.

Every layer, config, and manifest is identified by its SHA-256 digest. The digest is a hash of the content itself — change one byte, get a completely different digest.

This means:

Tag → Manifest digest → Config digest + Layer digests

Tag vs Digest

Tags are mutable pointers. Digests are immutable.

Tag	Digest
Mutable	Immutable
Human-friendly	Content-addressable
nginx:latest	sha256:abc123...

Example

# Pull by digest — always the exact same bytes
docker pull nginx@sha256:1234abcd...

# Pull by tag — could be different bytes tomorrow
docker pull nginx:1.25

# See the digest of a local image
docker images --digests nginx
# nginx  1.25  sha256:1234abcd...  2 weeks ago  187MB

Multi-Architecture Images

A single container image tag can support multiple CPU architectures by referencing a multi-architecture image manifest

    nginx:latest
        ├── amd64
        ├── arm64
        └── ppc64le

When you pull an image, the container runtime first downloads the manifest list, detects the host's CPU architecture, and automatically selects the appropriate image variant.

This allows developers to use the same image tag (for example, nginx:latest) across different hardware platforms without changing deployment manifests or Dockerfiles.

# On disk, containerd stores everything under its content store
ls /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/
# d4e5f6...  ← layer 1 compressed tarball
# a7b8c9...  ← layer 2 compressed tarball
# a1b2c3...  ← config JSON

Two images sharing a base layer reference the exact same bytes on disk — no duplication.

---

Step 3: Registry Pull Flow 📥

When a node needs an image it doesn't have, kubelet tells containerd to pull it:

Kubelet
  │  CRI: PullImage(nginx:1.25)
  ▼
containerd
  │
  ├─ 1. Resolve tag → registry API: GET /v2/nginx/manifests/1.25
  │     Headers: Authorization: Bearer <token>
  │     Response: manifest JSON + digest header
  │
  ├─ 2. Check local content store for each layer digest
  │     → Layer already present? Skip. Missing? Download.
  │
  ├─ 3. For each missing layer:
  │     GET /v2/nginx/blobs/sha256:<digest>
  │     Stream to /var/lib/containerd/.../ingest/<random>
  │     Verify: sha256(downloaded bytes) == digest from manifest
  │     Move to content store on success
  │
  └─ 4. Unpack config JSON, store in metadata DB (bbolt)

Image Pull Process

sequenceDiagram
    participant Runtime
    participant Registry

    Runtime->>Registry: Request Manifest

    Registry-->>Runtime: Manifest

    Runtime->>Registry: Download Layer 1

    Runtime->>Registry: Download Layer 2

    Runtime->>Registry: Download Layer 3

    Runtime-->>Runtime: Verify SHA256

    Runtime-->>Runtime: Store Locally

# Watch a pull happen with verbose output
containerd --log-level debug &
ctr images pull docker.io/library/nginx:1.25

# Or via crictl on a Kubernetes node
crictl pull nginx:1.25

Layer Deduplication at Pull Time

# Pull nginx — downloads 3 layers
ctr images pull nginx:1.25
# layer 1: sha256:abc...  31MB  pulled
# layer 2: sha256:def...  25MB  pulled
# layer 3: sha256:ghi...  1MB   pulled

# Pull nginx patch version — only pulls the changed layer
ctr images pull nginx:1.26
# layer 1: sha256:abc...  31MB  exists ← skipped, same bytes
# layer 2: sha256:def...  25MB  exists ← skipped
# layer 3: sha256:xyz...  1MB   pulled ← only this changed

---

Step 4: Snapshots — Assembling the Filesystem 💾

containerd does not directly mount OverlayFS. It uses a snapshotter — a pluggable interface for managing the filesystem state of a container.

The default snapshotter on Linux is overlayfs.

Snapshot Chain

Each image layer becomes a snapshot — an immutable point-in-time view of the filesystem up to that layer:

    💾 Snapshot 0: ubuntu base layer     ← lowerdir[2] 
         │
         ▼
    💾 Snapshot 1: + python installed    ← lowerdir[1]
         │
         ▼
    💾 Snapshot 2: + app code copied     ← lowerdir[0]
         │
         ▼
    💾 Active Snapshot (container)       ← upperdir (read-write)

# See snapshots on a node
ctr snapshots ls
# KEY                               PARENT          KIND
# sha256:abc...                                     Committed   ← layer 0
# sha256:def...        sha256:abc...                Committed   ← layer 1
# sha256:ghi...        sha256:def...                Committed   ← layer 2
# my-container-active  sha256:ghi...                Active      ← rw layer

The OverlayFS Mount 📂

Linux merges all layers. The container sees / instead of multiple layers.

When a container starts, containerd assembles the snapshot chain into an OverlayFS mount:

mount -t overlay overlay \
  -o lowerdir=/snapshots/ghi/fs:/snapshots/def/fs:/snapshots/abc/fs,\
     upperdir=/snapshots/active/fs,\
     workdir=/snapshots/active/work \
  /run/containerd/io.containerd.runtime.v2.task/<id>/rootfs

flowchart TD

    U["Writable Layer 📝"]
    U --> O["OverlayFS 📂"]

    L1["Layer 1 <br/> Ubuntu Layer 🐧<br/><br/>/usr/bin/bash"]
    L2["Layer 2 <br/> Python Layer 🐍<br/><br/>/usr/bin/python"]
    L3["Layer 3 <br/> Application Layer 🌐<br/><br/>/app/server.py"]

    L1 --> O
    L2 --> O
    L3 --> O

    O --> Container["Container FS 🗂️ <br/>/"]

What the container process sees:

   /
    │
    ├── /bin, /lib, /usr          ← from ubuntu layer (lowerdir[2])
    ├── /usr/local/lib/python3.11 ← from python layer (lowerdir[1])
    ├── /app/                     ← from app layer (lowerdir[0])
    └── /tmp/, /var/log/          ← writes go here (upperdir), ephemeral

Every read from a lower layer is zero-copy — the kernel serves it directly from the read-only snapshot. Every write triggers a copy-up to upperdir first.

---

Step 5: OCI Bundle — The Runtime's Input

Before calling runc, containerd produces an OCI bundle: a directory with two things:

/run/containerd/io.containerd.runtime.v2.task/<pod>/<container>/
├── rootfs/          ← the OverlayFS merged mount (the container's filesystem)
└── config.json      ← OCI Runtime Spec (namespaces, cgroups, mounts, process)

The config.json is generated by containerd from:

• The image config (env, cmd, user, working dir)
• The pod spec (resource limits, security context, volume mounts)
• The pod sandbox namespaces (NET, IPC, UTS to join)

{
  "ociVersion": "1.0.2",
  "process": {
    "user": {"uid": 1000, "gid": 1000},
    "args": ["nginx", "-g", "daemon off;"],
    "env": ["PATH=/usr/local/sbin:..."],
    "cwd": "/"
  },
  "root": {"path": "rootfs", "readonly": false},
  "namespaces": [
    {"type": "pid"},
    {"type": "mount"},
    {"type": "network", "path": "/proc/12345/ns/net"}  ← join pause container's NET ns
  ],
  "linux": {
    "resources": {
      "memory": {"limit": 536870912},
      "cpu": {"quota": 200000, "period": 100000}
    },
    "seccomp": {...},
    "maskedPaths": ["/proc/acpi", "/sys/firmware"]
  }
}

---

Step 6: Pod Sandbox — The Pause Container ⏸️

A pod is not a container.

A pod is a shared execution environment — a set of namespaces that multiple containers all join.

The holder of those namespaces is the pause container.

# See the pause container on any node
crictl ps | grep pause
# <id>   registry.k8s.io/pause:3.9   12h   Running   POD  my-pod

What the Pause Container Does

    kubelet creates pod sandbox
           │
           ▼
    containerd pulls pause:3.9 (if not cached)
           │
           ▼
    runc starts pause container with:
      - New NET namespace  → gets veth pair, assigned pod IP by CNI
      - New IPC namespace  → SysV IPC and POSIX shared memory scope
      - New UTS namespace  → pod hostname
           │
           ▼
    pause process runs: for(;;) pause();   ← does nothing, holds namespaces open
           │
           ▼
    CNI plugin wires pod IP into the NET namespace

The pause container runs an infinite loop doing nothing (pause() syscall). Its only job is to keep the namespaces alive.

If an app container crashes and restarts, the pod IP is preserved because the NET namespace belongs to the pause container, not to the app container.

# Confirm: pause container holds the NET namespace
PAUSE_PID=$(crictl inspect <pause-id> | jq -r '.info.pid')
ls -la /proc/$PAUSE_PID/ns/net
# lrwxrwxrwx ... net -> net:[4026532300]

APP_PID=$(crictl inspect <app-id> | jq -r '.info.pid')
ls -la /proc/$APP_PID/ns/net
# lrwxrwxrwx ... net -> net:[4026532300]   ← same inode — same namespace

---

Step 7: runc Creates the Container

containerd hands the OCI bundle to containerd-shim-runc-v2, which calls runc:

    runc create <container-id> --bundle /path/to/bundle
           │
           ├─ 1. Read config.json
           │
           ├─ 2. clone() — create new process in new namespaces
           │     CLONE_NEWPID  → new PID namespace (container gets PID 1)
           │     CLONE_NEWNS   → new MNT namespace
           │     (NET/IPC/UTS: join pause container's ns via "path" in config.json)
           │
           ├─ 3. Set up cgroup — write limits to /sys/fs/cgroup/...
           │     memory.max  = limits.memory
           │     cpu.max     = limits.cpu quota/period
           │
           ├─ 4. pivot_root — make the OverlayFS rootfs the new / inside MNT ns
           │
           ├─ 5. Mount special filesystems inside the new root:
           │     /proc  → new procfs (scoped to container's PID namespace)
           │     /sys   → sysfs (partially masked)
           │     /dev   → devtmpfs (minimal device set)
           │     /etc/resolv.conf → bind mount from kubelet
           │     /var/run/secrets/kubernetes.io/serviceaccount → projected volume
           │
           ├─ 6. Drop capabilities to configured set
           │     Apply seccomp filter
           │     Apply AppArmor profile
           │
           └─ 7. exec() → replace runc with nginx process
                  Now: nginx is PID 1 inside its own PID namespace
                  Sees: only OverlayFS filesystem, only pod's network, only its cgroup

---

Step 8: End-to-End — kubectl apply to Running Process

    User: kubectl apply -f deployment.yaml
             │
             ▼
    1.  API Server validates and writes Pod spec to etcd
    
    2.  Scheduler watches for unscheduled pods
        → Runs filter + score → assigns node
        → Writes pod.spec.nodeName to etcd
    
    3.  Kubelet on that node watches for pods assigned to it
        → Admits the pod (resource fits allocatable)
    
    4.  Kubelet calls containerd CRI: RunPodSandbox()
        → containerd starts pause container
        → CNI plugin creates veth pair, assigns pod IP
    
    5.  Kubelet calls containerd CRI: PullImage() for each container
        → containerd checks local content store
        → Downloads missing layers, verifies digests
        → Creates snapshot chain (OverlayFS)
    
    6.  Kubelet calls containerd CRI: CreateContainer()
        → containerd generates OCI config.json
        → config.json references pause container namespaces
    
    7.  Kubelet calls containerd CRI: StartContainer()
        → containerd-shim calls runc
        → runc: clone(), cgroup setup, pivot_root, exec()
        → nginx starts as PID 1 inside the container
    
    8.  Kubelet runs liveness/readiness probes
        → On success: sets pod.status = Running
        → API Server writes status to etcd
        → kubectl get pods shows Running

---

Image Layer to Filesystem: What You Actually See

    # Inspect the layer structure of a local image
    docker inspect python:3.11-slim | jq '.[0].RootFS.Layers'
    # [
    #   "sha256:layer1...",   ← debian:bookworm-slim base
    #   "sha256:layer2...",   ← apt-get install python3
    #   "sha256:layer3...",   ← pip defaults
    #   "sha256:layer4..."    ← python runtime setup
    # ]
    
    # Walk the actual OverlayFS mount for a running container
    CONTAINER_ID=$(docker ps -q --filter name=myapp)
    OVERLAY=$(docker inspect $CONTAINER_ID | jq -r '.[0].GraphDriver.Data.MergedDir')
    ls $OVERLAY
    # bin  dev  etc  home  lib  proc  root  sys  tmp  usr  var  app

    # See what a container has written to its rw layer
    UPPER=$(docker inspect $CONTAINER_ID | jq -r '.[0].GraphDriver.Data.UpperDir')
    find $UPPER -type f 2>/dev/null
    # ./var/log/nginx/access.log    ← nginx wrote this
    # ./tmp/somefile                ← app wrote this
    # → both disappear on container removal

---

Image Size vs Runtime Size

Metric	Meaning	How to check
Image size	Sum of all compressed layer tarballs	docker images nginx
Virtual size	Sum of all uncompressed layers (what OverlayFS exposes)	docker images --format "{{.VirtualSize}}"
Container size	Bytes written to upperdir (rw layer only)	docker ps -s
Shared layers	Bytes shared with other containers (from lowerdir)	docker system df -v

# How much disk is really being used?
docker system df
# TYPE            TOTAL    ACTIVE   SIZE      RECLAIMABLE
# Images          12       3        4.2GB     2.8GB (66% reclaimable)
# Containers      3        3        128MB     0B
# Local Volumes   5        2        9.1GB     5.2GB

# Remove dangling layers (untagged, unreferenced)
docker image prune

---

Linux Concepts: Image to Container

Stage	Linux Mechanism	What Happens
Layer storage	Content-addressable filesystem, SHA-256	Each layer stored once, identified by digest
Layer download	HTTP chunked transfer, splice(2)	Stream from registry to disk, verify digest
Layer extraction	tar + gzip, write(2)	Decompress tarball into snapshot directory
Snapshot chain	Hardlinks + directory tree	Each layer's snapshot points to parent
Filesystem assembly	OverlayFS mount()	Stacks snapshots into unified rootfs view
Copy-on-Write	Page fault → copy-up	Modified file copied from lowerdir to upperdir
Whiteout	Special .wh. files	OverlayFS hides deleted files from lower layers
Namespace creation	`clone(CLONE_NEWPID	CLONE_NEWNS)`
Namespace joining	setns(fd) with /proc/<pid>/ns/net	Container joins pause container's NET namespace
Root change	pivot_root(new_root, put_old)	Container's / becomes the OverlayFS merged dir
Resource limits	Write to /sys/fs/cgroup/	CPU quota and memory max applied before exec
Process start	exec(path, argv, envp)	runc replaced by container binary, becomes PID 1
Pod IP	Veth pair + bridge/VXLAN + CNI	pause container NET ns gets an IP from the CNI plugin

---

🧠 Interview Questions

Q: Two containers in the same pod share a network. How is this implemented at the kernel level? Both containers have a "network" entry in their OCI config.json pointing to /proc/<pause-pid>/ns/net. When runc calls setns(fd, CLONE_NEWNET) for each app container, they all join the same network namespace inode. From the kernel's perspective, they are the same network stack — same eth0, same IP, same routing table, same loopback.

Q: If I delete a file inside a container, does it free up disk space?

No — if the file was in a lower image layer. Deleting it writes a .wh.<filename> whiteout marker to the upperdir. The original bytes in the lowerdir are still on disk. Disk is only freed when the image itself is removed (docker rmi). The whiteout actually costs a tiny amount of space.

Q: I have 20 containers all running the same nginx:1.25 image. How much disk do the image layers use?

Exactly the same as one container — the layers are shared. All 20 containers share the same lowerdir snapshots. Each container has its own upperdir (written bytes only). docker system df shows shared vs unique bytes.

Q: A container restarts due to OOMKill. Does it get a new IP?

No. The IP belongs to the pause container's NET namespace, which was not restarted. The app container process died, runc re-creates it, and it rejoins the same NET namespace via setns(). The pod IP is stable across container restarts — only pod deletion and recreation triggers a new IP.

Q: What is the difference between docker commit and a Dockerfile layer?

Both produce a new image layer. docker commit snapshots the container's current upperdir into a new read-only layer appended to the image.

A Dockerfile RUN instruction does the same thing automatically for each build step. The result is identical OCI layers — docker commit is just manual layer creation. Both approaches produce layers with the same OverlayFS representation.

Q: How does Kubernetes know if an image pull failed vs a container crash?

Image pull failures surface as ErrImagePull → ImagePullBackOff in pod.status.containerStatuses[].state.waiting.reason. Container crashes appear as CrashLoopBackOff in the same field with exitCode populated. The distinction: image pull is a CRI PullImage failure; crash is a post-StartContainer process exit. kubectl describe pod shows Events with the exact error at each stage.

---

Related Posts

• Container Internals — Linux namespaces, cgroups, OverlayFS, and OCI spec in detail
• Kubernetes Pod Internals — pod lifecycle, pause container, and init containers
• Kubernetes Resource Allocation — how requests and limits map to cgroup values
• GPU Scheduling in Kubernetes — how nvidia-container-toolkit injects GPU devices at container start