Ansible

Ansible is Agentless Configuration Management

Terraform answers

"what infrastructure should exist?"

• VMs, networks, load balancers, GPU node pools.

Ansible answers:

"what should be installed and configured on the machines that already exist?"

• packages, users, config files, services, NVIDIA drivers, kubelet versions.

They get used together constantly: Terraform provisions the box, Ansible configures what runs on it.

flowchart TD

    Terraform["Terraform <br/> Provision infrastructure <br/> (VMs, VPCs, GPU node pools)"]
    Terraform-->|"hands off IPs / inventory"| Ansible["Ansible <br/> Configure the machines <br/> (packages, drivers, services)"]
    Ansible-->Running["Running, configured node <br/> ready for workloads"]

---

Why Ansible

Ansible automates the management of remote systems and controls their desired state.

Most configuration management tools before Ansible (Puppet, Chef) required an agent running on every managed node, phoning home to a central server on a schedule.

Ansible does neither. It's agentless and push-based:

flowchart TD

    Control["Control Node <br/> (your laptop / CI runner)"]
    Control-->|"SSH<br/>(or WinRM for Windows)"| Node1["Managed Node 1"]
    Control-->|"SSH"| Node2["Managed Node 2"]
    Control-->|"SSH"| Node3["Managed Node N"]

    Control-.->|"copies a small Python script<br/>over SSH, runs it, deletes it"| Node1

Ansible is Agentless, Push-Based

	Agent-based (Puppet, Chef)	Ansible
Requires software on managed node	Yes (agent daemon)	No — just SSH + Python
Execution model	Pull (agent polls the server)	Push (control node initiates)
Setup overhead	Install and register an agent everywhere	Nothing to install on targets
Real-time ad-hoc commands	Awkward	Native (ansible <host> -m ...)

Because there's no agent, a new node is manageable the moment it has SSH access and Python — which is exactly why it fits bare-metal GPU node bring-up so well:

• image the box
• get SSH working

and Ansible can take it from there.

Idempotency

Running ansible multiple times produces the same result.

The single most important property of a well-written playbook: running it twice produces the same end state, and the second run reports "nothing changed."

flowchart LR
    Run1["ansible-playbook site.yml <br/> (1st run)"]
    Run1-->Changed["changed=3 <br/> ok=5"]

    Run2["ansible-playbook site.yml <br/> (2nd run, same target)"]
    Run2-->NoOp["changed=0 <br/> ok=8"]

This is what makes a playbook safe to run in a cron job, in CI, or by a teammate without knowing exactly what state the target is currently in.

Ansible's built-in modules check current state before acting (apt checks if the package is already installed; service checks if it's already running) rather than blindly re-executing shell commands.

---

Core Concepts

flowchart LR
    subgraph Control["Control Node"]
        CLI["Ansible CLI"]
        Inventory["Inventory"]
        Playbooks["Playbooks"]
    end

    CLI --> SSH1
    CLI --> SSH2
    CLI --> SSH3

    SSH1 --> Host1["Managed Node 1"]
    SSH2 --> Host2["Managed Node 2"]
    SSH3 --> Host3["Managed Node 3"]

Concept	What it is
Inventory	The list of managed hosts, optionally grouped ([gpu_nodes], [web])
Module	A unit of work — apt, copy, service, user, nvidia_driver (custom)
Task	One module invocation with specific arguments
Playbook	A YAML file containing one or more plays
Play	A set of tasks mapped to a group of hosts
Role	A reusable, self-contained bundle of tasks, templates, and defaults
Handler	A task that only runs when notified (e.g. "restart nginx" after a config change)
Facts	Auto-discovered data about a host (OS, IP, CPU count, GPU presence)

💻 Control node

A system on which Ansible is installed.

The Control Node initiates connections to the managed nodes over SSH (or WinRM for Windows), reads the inventory, and executes tasks defined in playbooks.

The control node is responsible for:

• Reading the inventory
• Loading playbooks
• Connecting to remote hosts
• Executing Ansible modules
• Collecting task results

Example: Any Macbook with Ansible installed can work as Control Node

ansible

ansible-playbook

ansible-inventory

ansible-galaxy

📋 Inventory

An Inventory is a list of systems that Ansible manages.

Inventory on the control node describe host deployments to Ansible.

It tells Ansible:

• Which hosts exist
• How to connect to them
• How they are grouped

Example inventory:

    [web]
    web1.example.com
    web2.example.com
    
    [database]
    db1.example.com
    
    [production:children]
    web
    database

The inventory allows you to target groups instead of individual machines.

For example:

ansible web -m ping

This runs the ping module on every host in the web group.

At real cluster scale, inventory is usually dynamic

Usually a script or plugin (e.g. querying a cloud provider API, or NVIDIA Base Command Manager) generates the host list on the fly instead of a hand-maintained file.

[gpu_nodes]
dgx-01.cluster.local
dgx-02.cluster.local

[control_plane]
master-01.cluster.local

[gpu_nodes:vars]
ansible_user=ubuntu
nvidia_driver_version=550.90.07

🖥️ Managed node

A Managed Node is any remote machine controlled by Ansible.

Managed nodes:

• Do not require an Ansible agent
• Need Python installed (for most Linux modules)
• Must be reachable from the control node

Complete Workflow

sequenceDiagram
    participant User
    participant Control as Control Node
    participant Inventory
    participant Host as Managed Node 🖥️

    User->>Control: ansible-playbook site.yml
    Control->>Inventory: Read inventory
    Inventory-->>Control: Target hosts
    Control->>Host: SSH connection
    Control->>Host: Execute Ansible module
    Host-->>Control: Return results
    Control-->>User: Display task status

---

📒 Playbooks

A Playbook is a YAML file that defines the automation tasks Ansible should perform.

Anatomy of a Playbook

- name: Play Name
  hosts: web
  become: true

  vars:

  tasks:

  handlers:

Playbook Execution Flow

sequenceDiagram
    participant User
    participant Control Node
    participant Inventory
    participant Host

    User->>Control: ansible-playbook site.yml
    Control->>Inventory: Read inventory
    Inventory-->>Control: Target hosts
    Control->>Host: Execute Task 1
    Host-->>Control: OK
    Control->>Host: Execute Task 2
    Host-->>Control: Changed
    Control->>Host: Run Handler

Running a Playbook

ansible-playbook site.yml

Specify inventory

ansible-playbook \
    -i inventory.ini \
    site.yml

Dry run

ansible-playbook \
    --check \
    site.yml

📜 What is a Play?

A Play maps a group of hosts to a set of tasks.

📋 Tasks

A Task is a single unit of work.

Example

tasks:
  - name: Install Nginx
    ansible.builtin.apt:
      name: nginx
      state: present

Examples of tasks

• Install packages
• Create users
• Copy files
• Start services
• Configure firewalls

🧩 Modules

A unit of code or binary that Ansible runs on managed nodes.

Tasks execute modules.

- name: My first play
  hosts: myhosts
  tasks:
   - name: Ping my hosts
     ansible.builtin.ping:

   - name: Print message
     ansible.builtin.debug:
       msg: Hello world

There are many builtin modules/fucntions can be used to perform a task

    ansible.builtin.copy
    ansible.builtin.file
    ansible.builtin.user
    ansible.builtin.service

📝 Variables

Variables make playbooks reusable.

vars:
  package_name: nginx

Use

name: "{{ package_name }}"

Loops

Execute a task multiple times.

tasks:
  - name: Install packages
    ansible.builtin.apt:
      name: "{{ item }}"
      state: present
    loop:
      - git
      - curl
      - nginx

Conditionals

Run tasks only when conditions are met.

Example

- name: Install package
  ansible.builtin.apt:
    name: nginx
  when:
    ansible_os_family == "Debian"

🔧 Handlers

Handlers execute only when notified.

If nothing changes, the handler is not executed.

Example

tasks:

- name: Update Config
  ansible.builtin.copy:
    src: nginx.conf
    dest: /etc/nginx/nginx.conf
  notify:
    Restart Nginx

handlers:

- name: Restart Nginx
  ansible.builtin.service:
    name: nginx
    state: restarted

🏷️ Tags

Run specific tasks.

Example

tasks:
- name: Install Nginx
  tags:
    - install

Execute

ansible-playbook site.yml \
    --tags install

👤 Become

Run tasks as another user.

Usually root.

become: true

Equivalent to

sudo

ℹ️ Facts

Ansible automatically gathers system information.

Example

ansible_hostname

ansible_distribution

ansible_memory_mb

Disable Fact Gathering

gather_facts: false

---

🎭 Roles

A single flat playbook doesn't scale past a handful of tasks.

Roles package tasks, variables, templates, and handlers into a standard directory layout that any playbook can reuse:

roles/
└── nvidia_driver/
    ├── tasks/main.yml       # the actual steps
    ├── handlers/main.yml    # notify targets (e.g. reboot)
    ├── templates/           # Jinja2 config templates
    ├── defaults/main.yml    # default variable values (lowest precedence)
    └── vars/main.yml        # role-specific variables (higher precedence)

A playbook then just says:

- hosts: gpu_nodes
  roles:
    - nvidia_driver
    - kubelet_bootstrap

ansible-galaxy is the package manager for roles.

ansible-galaxy install nvidia.nvidia_driver pulls a published role the same way helm install pulls a chart, or terraform init pulls a provider.

---

Templates — Jinja2

Static config files rarely stay static across environments. Ansible renders Jinja2 templates (.j2 files), substituting variables per host:

# templates/kubelet-config.yaml.j2
maxPods: {{ max_pods_per_node }}
podCIDR: {{ pod_cidr }}
{% if gpu_node %}
featureGates:
  DevicePlugins: true
{% endif %}

- name: Render kubelet config
  template:
    src: kubelet-config.yaml.j2
    dest: /etc/kubernetes/kubelet-config.yaml
  notify: Restart kubelet

Same mechanism Helm uses for chart templates (see Helm: Kubernetes Package Manager) — one template, many rendered outputs, one per target.

---

Ansible Vault — Secrets

Playbooks routinely need secrets — registry credentials, API tokens, TLS keys.

Ansible Vault encrypts a file (or just specific variables) at rest so it's safe to commit alongside the rest of the playbook:

# Encrypt a file containing secrets
ansible-vault encrypt group_vars/gpu_nodes/secrets.yml

# Edit it later (decrypts, opens editor, re-encrypts on save)
ansible-vault edit group_vars/gpu_nodes/secrets.yml

# Run a playbook that needs vault-encrypted vars
ansible-playbook site.yml --ask-vault-pass

Encrypted content looks like this in the repo — safe to diff, safe to commit, useless without the vault password:

$ANSIBLE_VAULT;1.1;AES256
66386439653236336462626566653063336164663966303231363934653561363964363833313335
...

---

Ansible vs Terraform

The two aren't competitors so much as adjacent layers — but they overlap enough (both can technically create cloud resources; both are declarative-ish) that it's a very common interview comparison.

	Terraform	Ansible
Primary job	Provision infrastructure (what exists)	Configure infrastructure (what's installed/running)
State	Tracks state explicitly (terraform.tfstate)	Stateless — checks live system state on every run
Execution model	Plan → diff against state → apply	Sequential task execution against inventory
Language	HCL (declarative)	YAML (declarative-ish, but tasks run in order)
Idempotency mechanism	State file diffing	Per-module state checks
Typical scope	Cloud resources, networking, GPU node pools	OS packages, drivers, config files, service state
Agent required	No	No

In practice: Terraform creates the DGX node pool and the VPC; Ansible then installs the NVIDIA driver, container toolkit, and kubelet on top of what Terraform built. See Terraform IaC Concepts for the provisioning side of this pairing.

---

Essential Commands

# Ad-hoc command — no playbook needed
ansible gpu_nodes -m ping
ansible gpu_nodes -m shell -a "nvidia-smi --query-gpu=name,driver_version --format=csv"

# Run a playbook
ansible-playbook -i inventory.ini site.yml

# Dry run — show what would change without changing it
ansible-playbook site.yml --check --diff

# Target one host or one tag
ansible-playbook site.yml --limit dgx-01.cluster.local
ansible-playbook site.yml --tags "drivers"

# List all facts Ansible gathered about a host
ansible dgx-01.cluster.local -m setup

---

Key Takeaways

Concept	Summary
Agentless	Just SSH + Python on the target; nothing to install or maintain
Push-based	The control node initiates every run — no polling agent
Inventory	The list of managed hosts, static or dynamically generated
Playbook / Play / Task	Playbook contains plays; each play maps tasks to a host group
Module	The actual unit of work (apt, copy, service, ...) — checks state before acting
Idempotency	Re-running a playbook against an unchanged target reports zero changes
Role	Reusable, shareable bundle of tasks/templates/vars — the unit ansible-galaxy distributes
Handler	A task that only fires when notified by a change elsewhere in the play
Vault	Encrypts secrets at rest so they're safe to commit alongside the playbook
vs Terraform	Terraform provisions the infrastructure; Ansible configures what runs on it

Ansible's entire design point is removing the agent: no daemon to install, register, or lose contact with — just SSH and Python, which is exactly why it's the default choice for bringing a freshly-imaged bare-metal GPU node the rest of the way to "ready for workloads."

---

Interview Questions

Q: What makes Ansible "agentless," and why does that matter operationally? A: No persistent daemon runs on managed nodes — Ansible connects over SSH, copies a small Python module to the target, executes it, and removes it. Operationally this means there's nothing to install, upgrade, or lose contact with on thousands of nodes; a node is manageable the moment SSH and Python are available, which matters a lot during bare-metal bring-up.

Q: Explain idempotency in Ansible and why it's important. A: Idempotency means running the same playbook twice against an unchanged target produces zero additional changes the second time. Ansible modules check the current state of a resource before acting rather than blindly re-running commands — apt checks if a package is already at the target version, service checks if it's already running. This makes playbooks safe to schedule in cron or CI without knowing exactly what state a target is currently in.

Q: What is the difference between a module, a task, a play, and a playbook? A: A module is a single unit of work (apt, copy, service). A task is one module invocation with specific arguments. A play maps a list of tasks to a group of hosts from the inventory. A playbook is a YAML file containing one or more plays.

Q: How would you handle secrets in a playbook that gets committed to source control? A: Ansible Vault — encrypt the file (or specific variables within it) with ansible-vault encrypt, commit the encrypted content, and supply the vault password at run time (--ask-vault-pass, a vault password file, or an external secrets integration). The repo stays diffable and reviewable; the plaintext secret never touches source control.

Q: When would you reach for Ansible over Terraform, and when do they work together? A: Terraform answers "what infrastructure should exist" — it provisions VMs, networks, GPU node pools, and tracks that in state. Ansible answers "what should be installed and running on machines that already exist" — packages, drivers, config files, service state — and it's stateless, checking live system state on every run instead of diffing against a state file. The common pattern is Terraform provisioning the node, then handing its inventory off to Ansible to configure the OS, NVIDIA driver, container toolkit, and kubelet on top.

Q: How do dynamic inventories work, and why would you need one over a static inventory file? A: A dynamic inventory is a script or plugin that generates the host list at run time — querying a cloud provider's API, a CMDB, or a cluster manager like NVIDIA Base Command Manager — instead of a hand-maintained .ini file. It's necessary once nodes are added/removed frequently or autoscale, since a static file goes stale immediately in that environment.

Q: What's a handler, and how does it differ from a regular task? A: A handler is a task that only executes when explicitly notified by another task via notify, and only if that task reported a change. The canonical example is restarting a service after its config file changes — the restart shouldn't happen on every playbook run, only when the config actually changed. Handlers also only run once per play even if notified multiple times.

Q: How do roles improve on a single large playbook? A: Roles package tasks, handlers, templates, and default variables into a standard, reusable directory structure. Instead of one growing playbook, functionality is decomposed into independent, testable, shareable units (nvidia_driver, kubelet_bootstrap) that any playbook can compose by name, and that can be published and versioned via Ansible Galaxy — directly analogous to how Helm charts package Kubernetes manifests.

Q: How would you test a playbook change is safe before running it against production nodes? A: --check mode (dry run — reports what would change without changing it) combined with --diff (shows the actual before/after content of files that would be modified), scoped with --limit to a single canary host first, before running against the full inventory group.