Container Internals: What a Container Really Is

Container Internals 🐳

Containers do NOT run on different operating systems. They share the host operating system's kernel.

What is inside a container?

Container Image
├── Application
├── Libraries
├── Python/Java/Node
├── /bin
├── /etc
├── /usr
└── Root filesystem

How Container works

A container is not a VM. It is not a sandbox. It is not magic. It is a regular Linux process running with a restricted view of the system — created by combining three kernel features:

• namespaces,
• cgroups
• union filesystem.

Imagine an apartment building.

• Namespaces = Each family has its own apartment. They can't see inside other apartments.
• cgroups = The building manager limits electricity, water, and gas usage for each apartment.

A container needs both.

Understanding this is what lets you debug containers that behave differently from what you expect.

---

Container vs VM vs Process

	Bare Process	Container	VM
Kernel	Host kernel	Host kernel	Own kernel
Isolation	None	Namespace + cgroup	Hypervisor
Boot time	Milliseconds	Milliseconds	Seconds–minutes
Overhead	None	Near zero	CPU + RAM for guest OS
Filesystem	Host FS	OverlayFS layers	Virtual disk
Root access	Full	Restricted (capabilities)	Full

A container process is visible from the host:

# Start a container
docker run -d nginx

# See it from the host — it's just a process
ps aux | grep nginx
# root  1234  nginx: master process nginx -g daemon off

The difference: from inside the container, that process thinks it's PID 1. From the host, it's PID 1234.

---

The OS View: Three Kernel Features

1. Namespaces 🏢 — What the Container Can See

• Think of apartments in building. Each tenant live in isolation with own lift access, living area and storage.

A container gets its own set of Linux namespaces — a restricted, isolated view of system resources:

Namespace	What it isolates	Container gets its own
UTS(UNIX Time-Sharing)	SystemHostname	Hostname
PID	Process ID space	PID 1 (its own process tree)
MNT	Filesystem mount points	Filesystem tree (via OverlayFS)
NET	Network interfaces, routes	Network interface (shared with pod)
IPC	SysV IPC, POSIX shared memory	IPC objects
USER	UID/GID mappings	Root inside ≠ root outside
CGROUP	cgroup root view	Sees only its own cgroup tree

# See a container's namespaces from the host
PID=$(docker inspect --format '{{.State.Pid}}' my-container)
ls -la /proc/$PID/ns/
# lrwxrwxrwx mnt -> mnt:[4026532456]
# lrwxrwxrwx pid -> pid:[4026532457]
# lrwxrwxrwx net -> net:[4026532300]   ← same as pause container in a pod

PID namespace in action:

# Inside the container — process thinks it's PID 1
docker exec my-container ps aux
# PID   USER   COMMAND
#   1   root   nginx: master process

# From the host — same process is PID 1234
ps aux | grep nginx
# 1234  root   nginx: master process

The USER namespace — rootless containers:

# Container running as root inside, but maps to UID 1000 outside
cat /proc/$PID/uid_map
# 0  1000  1    ← container UID 0 = host UID 1000

A process that is "root" inside a USER namespace has no special privileges on the host. This is the foundation of rootless containers.

---

2. cgroups — What the Container Can Use

cgroups (control groups) enforce resource limits. Every container gets a cgroup subtree:

What resources can cgroups control?

Resource	Example
CPU	Maximum CPU usage
Memory	RAM limit
Disk I/O	Read/write bandwidth
Network (via extensions)	Traffic shaping
PIDs	Maximum number of processes
Huge Pages	Huge page allocation
Devices	Access to devices like GPUs

flowchart LR
K["🐧 Linux Kernel"]

    C["📦 Container Process"]

    PID["🆔 PID Namespace<br/>Process IDs"]
    NET["🌐 NET Namespace<br/>Network Stack"]
    MNT["💾 MNT Namespace<br/>Filesystem Mounts"]
    IPC["💬 IPC Namespace<br/>Shared Memory & Semaphores"]
    UTS["🖥️ UTS Namespace<br/>Hostname & Domain"]
    USER["👤 USER Namespace<br/>Users & Groups"]
    CG["⚙️ cgroups<br/>CPU • Memory • I/O"]

    K --> C

    C --> PID
    C --> NET
    C --> MNT
    C --> IPC
    C --> UTS
    C --> USER

    K --> CG
    CG -. Controls Resources .-> C

Examples: cgroup limit

/sys/fs/cgroup/
  memory/
    kubepods/
      burstable/
        pod<uid>/
          <container-id>/
            memory.limit_in_bytes    ← limits.memory
            memory.usage_in_bytes    ← current usage
  cpu/
    kubepods/
      burstable/
        pod<uid>/
          <container-id>/
            cpu.cfs_quota_us         ← limits.cpu (hard cap)
            cpu.shares               ← requests.cpu (weight)

cgroups v2 (unified hierarchy) — used in modern Linux kernels:

# cgroups v2 — single unified file per resource
cat /sys/fs/cgroup/kubepods.slice/memory.max
# 536870912   ← 512Mi limit

cat /sys/fs/cgroup/kubepods.slice/cpu.max
# 200000 100000   ← 200ms quota per 100ms period = 2 CPUs

# See what a container is actually using right now
cat /sys/fs/cgroup/kubepods.slice/<container>/memory.current

CPU throttling vs OOM kill:

• CPU — cgroup uses CFS (Completely Fair Scheduler). When a container exceeds cpu.cfs_quota_us, it is throttled — suspended until the next period. Never killed.
• Memory — when a container exceeds memory.limit_in_bytes, the kernel OOM killer fires immediately. The process is killed with signal 9.

# Check CPU throttling for a running container
cat /sys/fs/cgroup/kubepods.slice/<container>/cpu.stat
# usage_usec 123456789
# throttled_usec 45678   ← non-zero = being throttled
# throttled_periods 120

---

3. OverlayFS — What the Container Sees as its Filesystem

A container image is not a flat filesystem. It is a stack of read-only layers with a single read-write layer on top — assembled by OverlayFS.

Container filesystem (what the process sees)
│
├── Read-Write Layer    ← writable, ephemeral, deleted on container exit
├── Layer 4: app code  ┐
├── Layer 3: pip deps  │ read-only image layers
├── Layer 2: python    │ (shared between containers using same image)
└── Layer 1: ubuntu    ┘

# See the overlay mount for a running container
mount | grep overlay
# overlay on /var/lib/docker/overlay2/<id>/merged type overlay
#   (lowerdir=/var/lib/docker/overlay2/<l1>:<l2>:<l3>,
#    upperdir=/var/lib/docker/overlay2/<rw>/diff,
#    workdir=/var/lib/docker/overlay2/<rw>/work)

	OverlayFS dir	Purpose
	lowerdir	Stack of read-only image layers (colon-separated)
	upperdir	Read-write layer — all container writes go here
	workdir	OverlayFS internal workspace
	merged	The unified view the container process sees

Copy-on-Write (CoW): When a container modifies a file from a lower layer, OverlayFS copies it up to upperdir first, then modifies the copy. The original in lowerdir is unchanged — other containers sharing the layer are unaffected.

# Image layers are shared on disk — 10 nginx containers share the same lowerdir
docker inspect nginx:1.25 | jq '.[].RootFS.Layers'
# [
#   "sha256:abc...",   ← ubuntu base
#   "sha256:def...",   ← nginx binaries
#   "sha256:ghi..."    ← nginx config
# ]

Container Lifecycle

• Created – container is created but not started yet.
• Up – container is currently running.
• Paused – the container process was running, but Docker purposely paused the container.
• Exited – container has been terminated, usually because the container’s process was killed

End-to-End Lifecycle

    sequenceDiagram
    participant User
    participant Runtime as containerd
    participant Registry
    participant Snapshotter
    participant runc
    participant Kernel

    User->>Runtime: Run nginx
    Runtime->>Registry: Pull Image

    Registry-->>Runtime: Image Layers
    Runtime->>Snapshotter: Unpack Layers

    Snapshotter-->>Runtime: Root Filesystem
    Runtime->>runc: Create Container
    runc->>Kernel: Create Namespaces & cgroups
    Kernel-->>runc: Process Started
    runc-->>Runtime: Running

---

The OCI Standard

The Open Container Initiative (OCI) defines two specs that every container tool must implement:

1. Image Spec — What an image is

An OCI image is:

1. A manifest — JSON listing the config and layer digests
2. A config — JSON with env vars, entrypoint, cmd, working dir
3. A set of layer tarballs — each layer is a gzipped tar archive

// manifest.json
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "digest": "sha256:abc...",
    "mediaType": "application/vnd.oci.image.config.v1+json"
  },
  "layers": [
    {"digest": "sha256:def...", "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip"},
    {"digest": "sha256:ghi...", "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip"}
  ]
}

# Inspect an image's layers and config
docker manifest inspect nginx:1.25
docker image inspect nginx:1.25 | jq '.[].Config'

2. Runtime Spec — How to run a container

The OCI Runtime Spec defines a config.json describing the container to run:

{
  "ociVersion": "1.0.2",
  "process": {
    "user": {"uid": 0, "gid": 0},
    "args": ["nginx", "-g", "daemon off;"],
    "env": ["PATH=/usr/local/sbin:/usr/local/bin"],
    "cwd": "/"
  },
  "root": {"path": "rootfs", "readonly": false},
  "namespaces": [
    {"type": "pid"}, {"type": "network"}, {"type": "mount"}
  ],
  "linux": {
    "resources": {
      "memory": {"limit": 536870912},
      "cpu": {"quota": 200000, "period": 100000}
    }
  }
}

Any OCI-compliant runtime (runc, crun, youki) reads this spec and creates the container.

---

The Container Runtime Stack

Kubernetes talks to containers through a layered stack:

flowchart TB

    subgraph Kubernetes
        K["kubectl"]
        API["API Server"]
        KL["Kubelet"]
    end

    subgraph Runtime
        CRI["CRI (gRPC)"]
        CD["containerd <br/> high-level runtime"]
        SHIM["containerd-shim-runc-v2" ]
        RUNC["runc (OCI Runtime) <br/> low-level OCI runtime"]
    end

    subgraph Linux
        NS["Namespaces"]
        CG["cgroups"]
        OFS["OverlayFS"]
    end

    K --> API
    API --> KL
    KL --> CRI
    CRI --> CD
    CD --> SHIM
    SHIM --> RUNC

    RUNC --> NS
    RUNC --> CG
    RUNC --> OFS

containerd — manages image pulls, storage, snapshots, and delegates to runc for actual container creation. It is the CRI implementation that Kubernetes talks to.

runc — the reference OCI runtime. It reads config.json, calls clone() with the right namespace flags, sets up cgroups, mounts OverlayFS, and exec()s the container process.

# See containerd running on a node
systemctl status containerd

# List containers via CRI (not docker)
crictl ps

# List images via CRI
crictl images

# Pull an image via CRI
crictl pull nginx:1.25

---

From docker run to a Running Process

docker run nginx
       │
       ▼
1. Docker daemon looks up nginx:latest in local cache
   → If missing: pulls from registry (manifest → layers → extract)
       │
       ▼
2. containerd creates a snapshot (OverlayFS mount)
   → lowerdir = image layers
   → upperdir = fresh empty rw layer
   → merged  = container rootfs
       │
       ▼
3. containerd writes OCI config.json (namespaces, cgroups, mounts)
       │
       ▼
4. containerd-shim spawns runc
       │
       ▼
5. runc calls clone() with namespace flags:
   CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | CLONE_NEWUTS | CLONE_NEWIPC
       │
       ▼
6. runc sets up cgroup hierarchy, applies limits
       │
       ▼
7. runc mounts the OverlayFS rootfs inside the new MNT namespace
       │
       ▼
8. runc exec()s nginx as PID 1 inside the new PID namespace
       │
       ▼
9. nginx starts, sees itself as PID 1, sees only its OverlayFS filesystem

Complete Runtime Stack

    Application Request
            │
            ▼
    containerd
            │
            ├── Pull Image
            ├── Verify Digest
            ├── Create Snapshot
            ├── Generate OCI Bundle
            │
            ▼
    containerd-shim
            │
            ▼
    runc
            │
            ├── Create Namespaces
            ├── Configure cgroups
            ├── Mount Root Filesystem
            └── execve()
            │
            ▼
    Linux Kernel
            │
            ▼
    Container Process (PID 1)

Lifecycle Responsibilities

Stage	Component
Pull image	containerd
Verify image	containerd
Unpack layers	Snapshotter
Create snapshot	Snapshotter
Generate OCI bundle	containerd
Create namespaces	runc
Configure cgroups	runc
Start process	Linux Kernel
Monitor container	containerd / Kubelet
Restart (Kubernetes)	Kubelet
Remove resources	containerd

---

Container Security Primitives

Linux Capabilities

Traditional Unix: two privilege levels — root (all power) and non-root (almost nothing). Capabilities split root privilege into ~40 fine-grained units.

# What capabilities a process has
cat /proc/1/status | grep Cap
# CapPrm: 00000000a80425fb
# CapEff: 00000000a80425fb

# Decode it
capsh --decode=00000000a80425fb
# cap_chown, cap_net_bind_service, cap_kill, ...

In Kubernetes — drop everything, add only what you need:

securityContext:
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]   # nginx needs port 80

Common capabilities and why you rarely need them:

Capability	What it allows	Need it?
NET_ADMIN	Modify network interfaces, routes	Only for CNI plugins
SYS_PTRACE	Debug other processes	Only for debuggers
SYS_ADMIN	Mount filesystems, many other things	Almost never
NET_BIND_SERVICE	Bind to ports < 1024	Only if binding 80/443 as root
CHOWN	Change file ownership	Usually not needed

securityContext — Full Container Security

spec:
  securityContext:
    runAsNonRoot: true          # reject if image runs as root
    runAsUser: 1000             # force specific UID
    runAsGroup: 3000
    fsGroup: 2000               # volume files owned by this GID
    seccompProfile:
      type: RuntimeDefault      # default seccomp profile (blocks 300+ syscalls)

  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false   # no sudo, no setuid binaries
      readOnlyRootFilesystem: true      # can't write to container FS
      capabilities:
        drop: ["ALL"]
        add: ["NET_BIND_SERVICE"]

Seccomp — Syscall Filtering

The default seccomp profile blocks ~300 dangerous syscalls (ptrace, mount, kexec_load, reboot, etc.). Containers run with this by default in most managed clusters.

securityContext:
  seccompProfile:
    type: RuntimeDefault      # apply default seccomp profile
    # type: Localhost         # custom profile from node filesystem
    # type: Unconfined        # no filtering (dangerous)

# See what syscalls a container is making (for building custom profiles)
strace -p <pid> -e trace=all 2>&1 | grep -v ENOSYS

AppArmor

AppArmor enforces mandatory access control policies — what files, networks, and capabilities a process can access, regardless of its UID.

metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/nginx: runtime/default

---

Ephemeral Containers — Debugging Without SSH

Ephemeral containers are temporary containers added to a running pod for debugging. They share the pod's namespaces but do not appear in the pod spec and cannot be restarted.

# Add a debug container to a running pod
kubectl debug -it web-pod \
  --image=nicolaka/netshoot \
  --target=nginx          # share nginx's PID namespace

# Inside the debug container:
ps aux                    # sees nginx's processes
tcpdump -i eth0           # captures pod's network traffic
nsenter -t <nginx-pid> --mount -- ls /proc/1/root  # walk nginx's filesystem

# Debug a node directly
kubectl debug node/gpu-node-01 \
  -it --image=ubuntu
# Lands in a pod with the node's filesystem mounted at /host

---

Container Image Best Practices

Layer Ordering for Cache Efficiency

# Bad — code change invalidates everything below it
FROM python:3.11
COPY . /app               ← changes every commit
RUN pip install -r requirements.txt  ← reinstalls every commit

# Good — dependencies cached until requirements.txt changes
FROM python:3.11
COPY requirements.txt /app/
RUN pip install -r /app/requirements.txt   ← cached
COPY . /app                                 ← only layer that changes

Multi-Stage Builds — Minimal Runtime Image

# Stage 1: build
FROM golang:1.22 AS builder
WORKDIR /src
COPY . .
RUN go build -o /bin/app ./cmd/app

# Stage 2: runtime — no compiler, no source, no build tools
FROM gcr.io/distroless/static:nonroot
COPY --from=builder /bin/app /app
USER nonroot
ENTRYPOINT ["/app"]
# Final image: ~10MB vs 800MB builder image

Distroless Images

# gcr.io/distroless/static   — no shell, no package manager, no libc
# gcr.io/distroless/base     — adds glibc
# gcr.io/distroless/python3  — Python runtime only

# Result: drastically smaller attack surface
# No shell = no shell injection
# No package manager = no unexpected installs

Linux Concepts Used in Docker

Every Docker command maps directly to Linux kernel primitives. This table is the Rosetta Stone between Docker vocabulary and what is actually happening in the kernel.

Linux Concept	Kernel Feature	Docker / Container Equivalent	What it Does
Namespaces	clone() with CLONE_NEW* flags	Container isolation	Gives each container a private view of PID tree, network, mounts, hostname, IPC, and users
cgroups v1/v2	/sys/fs/cgroup/ hierarchy	--memory, --cpus, --gpus flags	Enforces resource limits; exceeding memory OOMKills, exceeding CPU throttles
OverlayFS	mount -t overlay	Image layers + read-write layer	Stacks read-only image layers; all container writes go to ephemeral upperdir
clone() syscall	unistd.h	docker run	Creates a new process in new namespaces — the container's PID 1
exec() syscall	unistd.h	ENTRYPOINT / CMD	Replaces the initial process image with the actual container binary
chroot / pivot_root	pivot_root(2)	Container filesystem root	Locks the container process into the OverlayFS-mounted rootfs
seccomp	prctl(PR_SET_SECCOMP)	--security-opt seccomp=	Allowlist/blocklist of syscalls the container process can make
Linux Capabilities	cap_set_proc(3)	--cap-add, --cap-drop	Splits root privilege into ~40 fine-grained units; containers start with a reduced set
veth pair	ip link add veth0 type veth	Container network interface (eth0 inside)	Virtual Ethernet cable — one end in container NET namespace, other end on host bridge
docker0 bridge	ip link add docker0 type bridge	Default Docker network	Linux bridge that connects all container veth pairs; provides LAN-like communication
iptables / nftables	Netfilter hooks	Port publishing (-p 8080:80)	NAT rules that translate host-port traffic into the container's NET namespace
/proc filesystem	VFS pseudo-filesystem	docker stats, docker top	Container metrics (CPU, memory, PIDs) are read from /proc/<pid>/ on the host
tmpfs	mount -t tmpfs	/dev/shm, --tmpfs mounts	In-memory filesystem; used for shared memory between processes in a pod
bind mount	mount --bind src dst	-v /host/path:/container/path	Makes a host directory visible at a path inside the container's MNT namespace
Unix sockets	AF_UNIX socket	Docker daemon API (/var/run/docker.sock)	CLI ↔ daemon communication; also kubelet ↔ containerd via CRI socket
inotify	inotify_add_watch(2)	Live-reload, config watchers	Watches filesystem events; container config managers use it to detect changes
signalfd / kill()	POSIX signals	docker stop (SIGTERM), docker kill (SIGKILL)	Graceful shutdown sends SIGTERM to PID 1; if timeout, SIGKILL forcibly ends it
setuid / setgid	setuid(2), setgid(2)	USER instruction, runAsUser	Drops root after startup; applies UID/GID mapping for USER namespaces
AppArmor	LSM hooks (security/apparmor/)	--security-opt apparmor=	Mandatory access control policy — what files, sockets, and capabilities a process may use
/dev/null, device files	Character device nodes	Stripped /dev inside container	Containers get a minimal /dev; GPU containers add /dev/nvidia* via Device Plugin
OOM killer	mm/oom_kill.c	Container restart on OOMKilled	Kernel selects the highest-score process (by oom_score_adj) to kill when memory is exhausted
Copy-on-Write (CoW)	mmap + page fault handler	Image layer sharing	Multiple containers share the same read-only image bytes; writes fork a private copy
/etc/resolv.conf	DNS resolver config	Container DNS	Docker bind-mounts a generated resolv.conf into the container's MNT namespace

---

🧠 Interview Questions

Q: What is the difference between a container and a VM?

• A container shares the host kernel — it is an isolated process using namespaces and cgroups. A VM runs its own kernel on a hypervisor.
• Containers have near-zero overhead and millisecond startup; VMs have stronger isolation but seconds to minutes startup and significant memory overhead.

Q: If two containers in different pods are on the same node, can they see each other's processes?

No. Each pod gets its own PID namespace (unless shareProcessNamespace: true). Container A cannot ps and see container B's processes. From the host, all processes are visible.

Q: A container writes a file to its filesystem. What happens to that file when the container restarts?

It is lost. Writes go to the upperdir of OverlayFS, which is the container's ephemeral read-write layer. On restart, containerd creates a fresh upperdir. Use a Volume to persist data across restarts.

Q: You need to run a legacy app that must bind to port 80. How do you avoid running it as root?

Add the NET_BIND_SERVICE capability and run as a non-root user:

securityContext:
  runAsUser: 1000
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]

Q: A container's memory usage keeps growing but never hits its limit. How do you investigate?

# Check actual usage vs limit
kubectl exec -it <pod> -- cat /sys/fs/cgroup/memory.current
kubectl exec -it <pod> -- cat /sys/fs/cgroup/memory.max

# Check if it's the process itself leaking or external pressure
kubectl exec -it <pod> -- cat /proc/meminfo | grep -E "MemFree|Cached|Buffers"

# Profile the process memory
kubectl exec -it <pod> -- valgrind --leak-check=full <cmd>

Q: What is the difference between CMD and ENTRYPOINT in a Dockerfile?

	ENTRYPOINT	CMD
Purpose	The executable to run	Default arguments
Override	--entrypoint flag	Append args after image name
Together	ENTRYPOINT ["nginx"] + CMD ["-g", "daemon off;"]	nginx gets -g "daemon off;"

Use ENTRYPOINT for the binary that must always run. Use CMD for overridable defaults.

---

Related Posts

• Kubernetes Pod Internals — how the pause container assembles namespaces into a pod from multiple containers
• Kubernetes Resource Allocation — requests, limits, QoS classes and how they map to cgroup values
• GPU Scheduling in Kubernetes — how nvidia-container-toolkit injects GPU devices into a container's namespaces at runtime
• Kubernetes Networking — how the container's NET namespace gets a pod IP and connects to the cluster network
• Cloud Native Observability — how metrics are scraped from inside containers via the NET namespace