CI/CD Pipelines: From Commit to Production

A GPU inference image that only ever gets built and deployed by hand doesn't scale past one engineer. CI/CD is the automation that turns "someone pushed code" into "the right, tested artifact is safely running in production" — without a human manually repeating the same steps every time.

flowchart TD
    Commit["git push"]
    Commit --> CI["Continuous Integration <br/> build + test"]
    CI --> Artifact["Artifact <br/> (image, chart, binary)"]
    Artifact --> CD["Continuous Delivery/Deployment <br/> release to environments"]
    CD --> Prod["Running in Production"]

---

CI vs Continuous Delivery vs Continuous Deployment

These three terms get used interchangeably in casual conversation, but they're distinct stages of automation maturity — and the difference is a very common interview question.

Term	What it guarantees
Continuous Integration (CI)	Every commit is automatically built and tested — you know within minutes if a change broke something
Continuous Delivery	Every commit that passes CI produces a release-ready artifact, deployable at any time — but a human still clicks "deploy"
Continuous Deployment	Every commit that passes CI is automatically deployed to production — no human gate at all

flowchart LR
    A["Continuous Integration"] --> B["Continuous Delivery"] --> C["Continuous Deployment"]
    A2["auto build + test"] -.-> A
    B2["+ auto-produce releasable artifact <br/> (manual deploy trigger)"] -.-> B
    C2["+ auto-deploy to prod <br/> (no manual gate)"] -.-> C

Most organizations stop at Continuous Delivery deliberately — a human approval gate before production is a feature, not a gap, especially for anything customer-facing or regulated.

---

The Pipeline — Stages

flowchart TD
    Source["Source <br/> (git push / PR)"]
    Source --> Build["Build <br/> compile, docker build"]
    Build --> Test["Test <br/> unit, integration, lint, security scan"]
    Test --> Package["Package <br/> push image to registry, version tag"]
    Package --> DeployStaging["Deploy: Staging"]
    DeployStaging --> Verify["Verify <br/> smoke tests, canary metrics"]
    Verify --> DeployProd["Deploy: Production"]

Each stage is a quality gate — a failure at Test should hard-stop the pipeline before anything reaches Package, and a failure at Verify should be able to automatically roll back DeployProd.

---

GitHub Actions

A workflow is triggered by an event (push, pull_request, schedule) and runs a sequence of jobs, each made of steps:

name: build-and-push

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t registry.internal/inference-service:${{ github.sha }} .

      - name: Run tests
        run: docker run --rm registry.internal/inference-service:${{ github.sha }} pytest

      - name: Push image
        run: |
          echo "${{ secrets.REGISTRY_TOKEN }}" | docker login -u ci --password-stdin registry.internal
          docker push registry.internal/inference-service:${{ github.sha }}

Every job runs on a fresh, ephemeral runner — nothing persists between runs unless explicitly cached, which is what makes CI reproducible: a pipeline that only passes because of leftover state from a previous run is a pipeline that will eventually fail mysteriously in production.

---

Jenkins — Declarative Pipelines

Jenkins predates GitHub Actions by over a decade and is still the default in a lot of enterprise and on-prem environments — including plenty of HPC/AI shops that already run their own infrastructure rather than relying on a SaaS CI. A Jenkinsfile is checked into the repo alongside the code, so the pipeline definition is versioned with the application it builds:

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh 'docker build -t registry.internal/inference-service:${GIT_COMMIT} .'
            }
        }
        stage('Test') {
            steps {
                sh 'docker run --rm registry.internal/inference-service:${GIT_COMMIT} pytest'
            }
        }
        stage('Push') {
            steps {
                sh 'docker push registry.internal/inference-service:${GIT_COMMIT}'
            }
        }
        stage('Deploy to Staging') {
            steps {
                sh 'helm upgrade --install inference-service ./chart --set image.tag=${GIT_COMMIT} -n staging'
            }
        }
    }
}

The agent directive picks which Jenkins worker runs the pipeline — on GPU-adjacent infrastructure this is often a specific label (agent { label 'gpu-runner' }) so build/test steps that need CUDA actually land on a node that has it.

---

Push-Based CD vs GitOps

There are two fundamentally different ways the last mile — actually updating what's running — gets done.

flowchart LR
    subgraph Push["Push-Based CD"]
        direction LR
        P1["CI pipeline"] -->|" kubectl apply / helm upgrade <br/> (CI has cluster credentials) "| P2["Kubernetes Cluster"]
    end

flowchart LR
    subgraph GitOps["GitOps"]
        direction LR
        G1["CI pipeline"] -->|" pushes new image tag "| G2["Git repo <br/> (Helm values / Kustomize)"]
        G3["ArgoCD"] -->|" watches, pulls "| G2
        G3 -->|" reconciles cluster to match repo "| G4["Kubernetes Cluster"]
    end

	Push-based CD	GitOps (ArgoCD, Flux)
Who has cluster credentials	The CI system	Only the in-cluster operator (ArgoCD)
Source of truth	The last thing CI ran	The Git repo, always
Drift detection	None — nobody notices manual kubectl edit	Automatic — operator continuously reconciles actual vs desired
Rollback	Re-run an old pipeline	git revert
Multi-cluster fan-out	CI script has to loop over clusters	Each cluster's ArgoCD watches the same repo

GitOps flips the trust model: instead of your CI system holding credentials to every cluster it deploys to, the cluster itself pulls from Git and reconciles continuously — the same control-loop pattern used everywhere else in Kubernetes ( see Kubernetes Informers & Controllers Explained), just applied one layer up, to " what should be running" instead of "what pods exist."

A minimal ArgoCD Application pointing at a Helm chart:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-service
spec:
  source:
    repoURL: https://github.com/org/gitops-repo
    path: charts/inference-service
    targetRevision: main
    helm:
      valueFiles:
        - values-prod.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: inference
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

selfHeal: true is the key line — if someone manually edits a Deployment in the cluster, ArgoCD reverts it back to match Git on the next reconciliation pass, the same way any Kubernetes controller reverts drift from desired state.

---

Real Example: Deploying a GPU Inference Service

Tying the whole pipeline together, end to end:

flowchart TD
    Dev["Engineer pushes code"]
    Dev --> CI["CI: build + test + push image <br/> registry.internal/nim-service:abc123"]
    CI --> PR["CI opens a PR bumping <br/> image.tag in gitops-repo"]
    PR --> Merge["Merged to main"]
    Merge --> Argo["ArgoCD detects change, syncs"]
    Argo --> Cluster["Helm upgrade rolls out <br/> new NIM pods on GPU nodes"]
    Cluster --> DCGM["DCGM / Prometheus <br/> watch rollout health"]

CI never touches the cluster directly — its entire job ends at "produce a tested image and propose the version bump in Git." Everything from that point on is the cluster reconciling itself against Git, observed through the same Cloud Native Observability stack already watching the rest of the cluster.

---

Key Takeaways

Concept	Summary
CI	Every commit auto-built and tested
Continuous Delivery	Every passing commit produces a release-ready artifact; deploy is a manual trigger
Continuous Deployment	Every passing commit auto-deploys to production, no manual gate
Pipeline stages	Build → Test → Package → Deploy → Verify, each a quality gate
GitHub Actions / Jenkins	Two common CI engines; Jenkinsfile is versioned pipeline-as-code, same as a GitHub Actions workflow file
Push-based CD	CI holds cluster credentials and applies changes directly
GitOps	Cluster pulls from Git and self-heals; CI never touches the cluster, only Git
selfHeal	ArgoCD reverts manual drift back to match the Git-declared state automatically

The real shift GitOps makes isn't tooling, it's where trust lives: push-based CD means every CI job is a credentialed path into your cluster; GitOps means the only thing with cluster credentials is the reconciler already running inside it, and Git becomes the single audit trail for every change that ever reached production.

---

Interview Questions

Q: What's the difference between Continuous Delivery and Continuous Deployment?

A: Both require every commit to pass through CI (auto build + test) and produce a release-ready artifact. Continuous Delivery stops there — a human still triggers the actual production deploy. Continuous Deployment removes that manual gate entirely, so a passing commit ships to production automatically. The distinction is purely about whether there's a human approval step before production.

Q: Why would a team deliberately choose GitOps over a CI pipeline that runs kubectl apply directly?

A: Credential security and drift detection. With push-based CD, the CI system needs cluster credentials, which means every CI job — including third-party actions and dependencies — is a potential path into production. With GitOps, only the in-cluster operator (ArgoCD/Flux) has cluster credentials; CI's job ends at proposing a change in Git. GitOps also continuously reconciles actual state against the Git-declared state, so a manual kubectl edit gets automatically reverted instead of silently drifting unnoticed.

Q: How does ArgoCD's selfHeal work, and why does it matter operationally?

A: With selfHeal: true, ArgoCD doesn't just sync on a Git change — it continuously compares the live cluster state to what's declared in Git, and reverts any manual, out-of-band change back to match Git on the next reconciliation pass. Operationally this means Git is truly the single source of truth: someone can't quietly patch a Deployment in an emergency and have that fix silently persist forever without it being reflected in the repo.

Q: What should happen if the Test stage fails in a pipeline — and why does that matter more than a Build failure?

A: The pipeline should hard-stop and never reach the Package/Deploy stages — nothing that failed its test suite should ever get tagged as a deployable artifact or pushed to a registry. A build failure just means the code doesn't compile; a test failure means the code compiles but is provably wrong, which is exactly the class of bug CI exists to catch before it reaches an environment where it's expensive to find.

Q: How would you design a CI/CD pipeline for a GPU-accelerated service differently from a typical web service?

A: The build/test stage needs to run on a runner that actually has GPU access if tests exercise CUDA code paths — a plain CPU-only runner can build the image but can't validate GPU-dependent behavior. The deployment side benefits from GitOps (ArgoCD/Flux) syncing a Helm chart so the rollout goes through the same GPU Operator / device plugin scheduling path as any other GPU workload, and rollout health should be verified against GPU-specific signals (DCGM metrics, MIG allocation) rather than just HTTP health checks, since a pod can be "Running" and still be GPU-starved.

Q: What's a rollback mechanism under GitOps compared to a traditional push-based pipeline?

A: Under GitOps, rollback is a git revert of the commit that bumped the image tag or changed the manifest — the operator picks up the reverted state on its next sync and reconciles the cluster back to it automatically. Under push-based CD, rolling back typically means re-running an older pipeline job or manually re-applying a previous manifest, which depends on that old pipeline run (and its artifacts) still being available and valid.

Q: Why is it important that CI runners are ephemeral rather than long-lived, reused machines?

A: A pipeline that only passes because of leftover state from a previous run (a cached dependency that was manually patched, a stale config file) is a pipeline that will eventually fail unpredictably once that specific runner's state changes or a job lands on a different machine. Ephemeral runners force every build to be reproducible from the repo and declared dependencies alone, which is exactly the property CI is supposed to guarantee.