Deploying Agentic AI to Production

Deploying Agentic AI to Production

Deployment Strategies

1. Shadow Deployment 👤

Deploy a new agent version alongside the production version and mirror real production traffic to it without exposing responses to users.

The shadow system receives the same requests as production but its responses are discarded.

Purpose:

• Validate agent behavior on real traffic
• Compare latency and cost
• Detect hallucinations
• Verify tool integrations
• Measure reasoning quality
• Test new prompts, models, or workflows safely

Architecture

flowchart TD

    User --> ProductionAgent

    User -. Mirrored Traffic .-> ShadowAgent

    ProductionAgent --> Response


    ProductionAgent --> Metrics
    ShadowAgent --> Metrics

    Metrics --> Dashboard

    ShadowAgent -. Discard Output .-> Trash[(Ignored)]

Benefits

• Safe evaluation: Real-world testing with Zero user impact
• Performance benchmarking across various models
• Validation of tool integrations

Agentic AI Example

Current Production:

GPT-4.1 + ReAct

Shadow Deployment:

Llama Nemotron + ReWOO

Both receive identical requests.

Compare:

• Task Success Rate
• Hallucination Rate
• Cost Per Request
• Tool Usage Accuracy
• Latency

Users only see production responses.

Shadow vs Canary

Canary	Shadow
Real users see responses	Users never see responses
Limited production exposure	No production exposure
Tests business impact	Tests technical behavior
Can affect users	Zero user impact
Lower infrastructure cost	Higher infrastructure cost
Used before full rollout	Used before canary

Production Flow

flowchart TD

    Code--> OfflineEvaluation --> ContainerBuild--> ShadowDeployment--> CanaryDeployment--> FullProduction

---

2. Canary Deployment 🐤

Gradually expose a new agent version to a small percentage of real users before rolling it out to everyone.

Unlike a shadow deployment, users actually receive responses from the new version.

The goal is to validate production behavior while limiting risk.

A canary deployment minimizes blast radius.

flowchart TD

    Traffic
    AgentV1[Agent 1.0]
    AgentV2[Agent 2.0]
    User1["95% User <br/> 🐦 Stable Model"]
    Users2["5% User <br/> 🐤 Canary Model"]


    Traffic -->|95%| AgentV1
    Traffic -->|5%| AgentV2

    AgentV1 --> User1
    AgentV2 --> Users2

Performance Metrics

User Satisfaction & Reliability

• Task Completion Rate
• Tool Success Rate
• Error Rate

Quality

• Hallucination Rate
• Answer Accuracy
• Reasoning Quality

Performance

• Latency
• Throughput
• GPU Utilization

Cost

• Tokens Per Request
• Inference Cost
• Infrastructure Cost

A common exam scenario:

A company wants to test a new multi-agent workflow on 5% of users while monitoring hallucination rates and latency before a full rollout.

Answer: 🐤 Canary Deployment

Automated Promotion

Many production systems automatically promote a canary when metrics pass thresholds.

flowchart TD

    A[Canary Deployment]

    B{Metrics Healthy?}

    C[Increase Traffic]

    D[Rollback]

    A --> B

    B -->|Yes| C
    B -->|No| D

Example:

Latency < 2 seconds

Hallucination Rate <= Production

Success Rate >= Production

Promote automatically.

Automated Rollback

If problems appear:

Latency Spike
Hallucination Increase
Tool Failures

Traffic immediately returns to the previous version.

flowchart TD

    Traffic --> Canary

    Canary --> Failure

    Failure --> Rollback

    Rollback --> StableVersion

Examples

• Model Canary: GPT-4.1 --> GPT-5
• Prompt Canary: Prompt V1 --> Prompt V2
• RAG Canary: Old Retrieval Pipeline --> New Retrieval Pipeline
• Agent Architecture Canary : ReAct --> ReWOO

---

3. 🧪 A/B Testing

An experimentation technique used to compare variants.

Purpose:

• Measure user behavior
• Compare outcomes
• Optimize conversions
• Validate hypotheses

flowchart TD
    Request --> Split{"Experiment <br/>Group?"}

    VariantA[Variant A<br/>Agent GPT-4.1]
    VariantB[Variant B<br/>Agent GPT-5]

    Split -->|50%| VariantA 
    Split -->|50%| VariantB

A/B testing is not a subtype of Canary.

Think of them as siblings:

    Traffic Splitting
    │
    ├── 🐤 Canary Deployment
    │   └── Risk Reduction
    │
    └── 🧪 A/B Testing
        └── Experimentation

Canary vs A/B testing

Aspect	Canary Deployment	A/B Testing
Goal	Reduce deployment risk	Compare alternatives
Traffic Split	Usually unequal (95/5, 90/10)	Usually equal (50/50, 30/30/40)
Success Criteria	Stability, latency, errors	Business or quality metrics
End Result	Promote or rollback	Choose best variant
Focus	Release strategy	Experimentation strategy

Feature Flag 🚩

Feature flags are often used to implement A/B testing.

We can hide an unstable/ new feature from user and

• Measure user behavior
• Compare outcomes
• Optimize conversions
• Validate hypotheses

Flow

flowchart TD

    PromptV1["New App UI"]
    PromptV2["Legacy App UI"]

    Testing[Staging Env]
    Production[Production Env]
    

    Flag{"Feature <br/> Flag enable 🚩"}
    User --> Flag

    Flag -->|No| Production--> PromptV2
    Flag -->|yes| Testing-->PromptV1

Help with developing a new feature.

---

4. Blue-Green Deployment 🔵🟢

Maintain two identical production environments and switch traffic between them during releases.

Only one environment serves users at a time.

🔵 Blue  = Current Production 🚀
🟢 Green = New Release Standby ⛔

Release 1
🟢 Green  = Current Production 🚀
🔵 Blue = New Release Standby ⛔

Release 2
🔵 Blue  = Current Production 🚀
🟢 Green = New Release Standby ⛔

Release 3
🟢 Green  = Current Production 🚀
🔵 Blue = New Release Standby ⛔

flowchart TD

    BlueProd["🔵 Blue  = Production 🚀"]
    BlueStand["🔵 Blue  = Standby ⛔"]


    GreenProd["🟢 Green   = Production 🚀"]
    GreenStand["🟢 Green   = Standby ⛔"]

     BlueProd -->|SwitchTraffic| BlueStand --> |Green Deployment| GreenProd

    GreenProd-->|SwitchTraffic| GreenStand--> |Blue Deployment| BlueProd

When the new version is ready:

1. Deploy to Green
2. Validate functionality
3. Switch traffic
4. Monitor
5. Roll back instantly if needed

Architecture

flowchart TD

    A[🔵 Blue Live]

     B[Deploy New Version to 🟢 Green]

    B --> C[Validation & Smoke Tests]

    C --> D[Switch Traffic]

    D --> E[🟢 Green Live]

    E --> F{Issue Detected?}

    F -->|Yes| G[Switch Back to 🔵 Blue]
    
    G --> A

    F -->|No| H[Keep Green Live]

Loop

flowchart TD

    A[🔵 Blue Live]

    A --> B[Deploy New Version to 🟢 Green]

    B --> C[Validate]

    C --> D[Switch Traffic]

    D --> E[🟢 Green Live]

    E --> F{Healthy?}

    F -->|No| G[Switch Back to Blue]
    G --> A

    F -->|Yes| H[Green Becomes Blue]

    H --> J[Deploy Next Version to Idle Environment]

    J --> C

Benefits

• Near-zero downtime
• Simple rollback
• Full production validation
• Easy version comparison
• Predictable deployment process

Drawbacks

• Double infrastructure cost
• Duplicate databases may be needed
• More operational complexity
• Not ideal for validating unseen production traffic patterns

Blue-Green vs Canary vs Shadow

Strategy	User Exposure	Traffic Distribution	Rollback Speed	Primary Goal
🔵🟢 Blue-Green	100% after switch	All traffic to one environment	Instant	Safe release
🐤 Canary	Small percentage	Split traffic	Fast	Gradual rollout
👤 Shadow	None	Mirrored traffic	Not needed	Validation

---

Final Words

Full Rollout

→ All users

Shadow Deployment

• → No user exposure
• → Mirrored production traffic: Real traffic
• → Responses discarded

Canary Deployment

• → Real traffic
• → Small percentage of users
• → Gradual rollout

Blue-Green

• → Two identical environments
• → Full traffic switch
• → Instant rollback

---

Related Posts

• Deploying Agents at Scale — the deployment step this ongoing operations practice follows
• CI/CD Pipelines: From Commit to Production — the general CI/CD pipeline concepts this agent-specific DevOps builds on