Tech Stack
Summary
Simulates up to 10,000 Kubernetes nodes with 80,000 GPUs on a single laptop using KWOK and ClusterLoader2 — benchmarked to 142.8 pods/sec scheduling throughput and 0.530s p99 latency at scale.
What I Built
Project Overview
Testing Kubernetes at GPU-cluster scale normally requires hundreds of real nodes and thousands of dollars of hardware. GhostFleet collapses that barrier: it simulates up to 10,000 nodes with 80,000 GPUs on a single laptop by combining KWOK (Kubernetes Without Kubelet) for lightweight node emulation with ClusterLoader2 for official Kubernetes density benchmarking.
"A 10,000-node GPU cluster with nobody aboard."
The real Kubernetes control-plane components — kube-apiserver, etcd, kube-scheduler — run as normal. Only the worker nodes are simulated. This means scheduler decisions, API server load, etcd write amplification, and control-plane bottlenecks all behave exactly as they would at production scale.
Key Features
Hyperscale Simulation on a Laptop
Emulate clusters that would normally cost thousands per hour in cloud compute — useful for studying Kubernetes limits before committing to real infrastructure.
Real Control Plane, Fake Nodes
KWOK simulates kubelet behaviour for each virtual node without running actual container workloads. The scheduler sees 8,000 nvidia.com/gpu resources and makes real placement decisions.
ClusterLoader2 Integration
Uses the official Kubernetes SIG-Scalability benchmark tool to run reproducible density tests — the same tooling used to validate upstream Kubernetes releases.
Prometheus Observability
Custom metrics and Grafana dashboards capture control-plane latency, API server request rates, etcd operation counts, and scheduler throughput throughout benchmark runs.
Automated Reporting
GitHub Actions publishes benchmark results to GitHub Pages after each run — shareable, versioned performance reports with no manual effort.
Benchmark Results
Live results published at hiteshsahu.github.io/ghostfleet — auto-updated by GitHub Actions after each experiment run.
| Nodes | GPUs | Pods | Scheduling Throughput | p99 Latency | etcd Size |
|---|---|---|---|---|---|
| 10,000 | 80,000 | — | 13.7 pods/sec | — | 578.2 MB |
| — | — | — | 142.8 pods/sec | 0.530s | — |
6 experiment runs conducted on July 19, 2026, ranging from small-scale validation to the 10,000-node peak. The highest throughput run (142.8 pods/sec, 0.530s p99) demonstrates sub-second scheduling latency at GPU-cluster scale using only simulated hardware.
These numbers directly answer the question: "How does the Kubernetes control plane behave when scheduling across an 80,000-GPU fleet?" — without provisioning a single real node.
Why It Matters
| Without GhostFleet | With GhostFleet |
|---|---|
| Need a real 10,000-node cluster to test at scale | Simulate it locally in minutes |
| Control-plane behaviour at 80,000 GPUs is unknown until production | Reproduce it before provisioning a single node |
| Benchmarks are manual and not reproducible | ClusterLoader2 + CI produces consistent, versioned results |
| Expensive cloud spend to validate scheduler changes | Zero hardware cost — 142.8 pods/sec measured on a laptop |
For teams building AI infrastructure on Kubernetes — DGX Cloud, on-prem GPU clusters, or cloud GPU node pools — understanding how the control plane behaves under 1,000-node GPU workloads is critical before provisioning real hardware.
Technical Highlights
KWOK Node Emulation
Each virtual node registers with the API server with configurable capacity (nvidia.com/gpu: 8) and responds to kubelet-level lifecycle events — making it indistinguishable from a real node from the scheduler's perspective.
GPU Resource Simulation
Virtual nodes advertise NVIDIA GPU resources. Pods requesting nvidia.com/gpu are scheduled, placed, and tracked by the real Kubernetes scheduler — exercising GPU-aware scheduling logic at scale.
Benchmark Reproducibility
ClusterLoader2 test configs define pod density, churn rate, and scheduling pressure in code — results are deterministic across runs and comparable across Kubernetes versions.
Technology Stack
Cluster Simulation: KWOK (Kubernetes Without Kubelet)
Benchmarking: ClusterLoader2 (Kubernetes SIG-Scalability)
Observability: Prometheus, Grafana
Runtime: Docker / Podman
Automation: GitHub Actions, Shell
Reporting: GitHub Pages
Tools: kubectl, jq, Go