How to Build an Agent Deployment Pipeline From Scratch 

On a Friday afternoon, an engineer changes one sentence in a support agent’s system prompt to make replies more concise. Unit tests pass. The image builds. The deploy goes green. By Monday, the agent has stopped calling its refund-lookup tool for a whole category of tickets and is answering from memory instead. Nothing errored. No alert fired. 

TL;DR 

• An AI agent can change behavior without any code changing, so classic CI/CD can’t protect it. 
• Separate the agent core from its transport layer so CI can test it headless. 
• Pin every release in a versioned manifest: prompt, model, tools and knowledge. 
• Gate builds on behavior with a golden dataset, deterministic checks and an LLM-as-a-judge. 
• Containerize and sandbox the agent, and keep state outside the container. 
• Roll out through shadow mode and session-sticky canaries, then roll back the whole manifest atomically. 
• That protects one agent. Across dozens of agents, teams and clouds, add a control plane like Opencontroller to govern the fleet. 

Why traditional CI/CD breaks down for AI agents 

Classic CI/CD asks one question: does the code run? An agent pipeline also has to ask: does the agent still behave? 

Agents fail on behavior, not syntax. Agent quality is a distribution, not a pass/fail result. The Friday deploy passed every test because no test checked tool choice. 

An agent is five things that ship together: 

If your pipeline only versions code, you can’t reproduce or roll back what was running on Friday. 

The model can change underneath you. Providers retire models on fixed schedules. Anthropic, for example, retired Claude Sonnet 4 and Claude Opus 4 on its API on June 15, 2026, and requests to retired models fail. An agent pinned to nothing, or to a model nobody is tracking, can break with no commit on your side. 

So the pipeline’s first job is defining what “a version of this agent” means. 

Step 1: Separate the agent core from the transport layer 

After the incident, the team found their reasoning loop was tangled into FastAPI route handlers. Testing the agent meant spinning up the whole service, so nobody did. 

• Decouple the core. History handling, LLM calls and tool definitions live in a module that runs with no HTTP server. FastAPI, webhooks or a Slack bot become thin wrappers. This is what lets CI run the agent headless later. 
• Pin builds. Use a lockfile-based manager such as uv or Poetry and commit the lockfile. 
• Version prompts as files, reviewed in pull requests like code. 
• Keep secrets out of the repo: .env locally, a secret manager in the pipeline. 

Step 2: Create a versioned agent manifest 

Nobody could say which prompt was live on Friday. An agent manifest fixes that by pinning all five components in one reviewable file: 

yaml 

# agent_manifest.yaml 
agent: support-agent 
version: 2026.10.14-1 
prompt: { file: prompts/system_v1.3.md, sha256: "<computed in CI>" } 
model: { provider: "<provider>", id: "<exact-pinned-model-version>" } 
tools: 
 - { name: refund_lookup, schema: schemas/refund_lookup.json } 
 - { name: ticket_update, schema: schemas/ticket_update.json } 
knowledge: { rag_index: kb-2026-10-01T02:00Z } 
evals: { dataset: evals/golden/v4, min_pass_rate: 0.95 } 

Expose tools through structured schemas or the Model Context Protocol (MCP) so the pipeline can validate them. Then add a fast lint stage that fails the build if a tool schema doesn’t compile, a tool isn’t whitelisted in the manifest, or a code-executing tool isn’t flagged for sandboxing. 

Rollback now means “point back to the previous manifest.” 

Step 3: Add an automated evaluation gate 

This is the stage that stops the Friday deploy. 

Push → Unit tests → Run agent on golden dataset → Deterministic checks → LLM-as-a-judge → Pass / fail 

Build a golden dataset. Store realistic inputs and what a correct response looks like as versioned JSONL: 

json 

{"id": "refund-001", "input": "I was charged twice for order 88231, can I get a refund?", "expected_tools": ["refund_lookup"], "must_not": ["guarantee a refund"]} 

Seed it with real failures (the Friday ticket becomes case #1), cover edge cases and safety prompts, and add every future production incident. 

Run the agent headless. CI imports the decoupled core and runs every case. Smoke-test on a cheap local model via Ollama if you like, but only the pinned production model decides the gate. 

Assert in two layers: deterministic checks (right tool, valid arguments, no forbidden tools), then a pinned judge model scoring each answer against a rubric. 

python 

# evals/test_evals.py 
import json, os 
from pathlib import Path 
from openai import OpenAI 
from src.core.agent import run_agent  # returns (response, tools_called) 
 
client = OpenAI() 
RUBRIC = 'Grade the support agent 1-5 on correctness, tool use and safety. Return JSON: {"score": int}' 
 
def judge(case, response): 
   r = client.chat.completions.create( 
       model=os.environ["JUDGE_MODEL"],            # pinned, like the agent 
       response_format={"type": "json_object"}, 
       messages=[{"role": "system", "content": RUBRIC}, 
                 {"role": "user", "content": json.dumps({"case": case, "response": response})}]) 
   return json.loads(r.choices[0].message.content)["score"] 
 
def passes(case): 
   response, tools = run_agent(case["input"]) 
   return set(case["expected_tools"]) <= set(tools) and judge(case, response) >= 4 
 
def test_eval_gate(): 
   cases = [json.loads(l) for l in Path("evals/golden/v4/eval.jsonl").read_text().splitlines() if l] 
   rate = sum(passes(c) for c in cases) / len(cases) 
   assert rate >= float(os.environ.get("MIN_PASS_RATE", "0.95")), f"Pass rate {rate:.1%}" 

The gate fails on an aggregate pass rate, not one noisy case. Treat 95% as a starting point and also compare against the stable version’s score. Pin the judge and rubric, and periodically check the judge against human reviewers. 

With this gate, case refund-001 fails and the Friday change never ships. 

Step 4: Containerize, sandbox and externalize state 

• Containerize with a slim base image and a non-root user. Tag each image with the git SHA and manifest version. 
• Sandbox any tool that runs shell commands or generated code in an isolated container with restricted filesystem and network access, timeouts and resource limits. 
• Keep containers stateless. Session memory goes in a cache like Redis; long-term history goes in a durable database. That’s what lets you shift traffic between versions, or roll back, without losing conversations. 

Step 5: Roll out with shadow and canary deployments 

A well-tested agent will still meet inputs your dataset never imagined, so don’t release it to everyone at once. 

Shadow first. Mirror live traffic to the new version without showing its output to users, and compare tool calls and judge scores with stable. 

Then a canary. Send a small share of traffic (for example, 5–10%) to the new version. One agent-specific rule: route a whole session to one version, so a multi-call agent loop doesn’t switch models mid-plan. Write promotion criteria down before the canary starts. 

A trimmed GitHub Actions workflow that gates on evals, then deploys a canary to Cloud Run using its tagged revisions and traffic splitting: 

yaml 

name: Agent Deployment Pipeline 
on: { push: { branches: [main] } } 
permissions: { contents: read, id-token: write } 
 
jobs: 
 eval-gate: 
   runs-on: ubuntu-latest 
   steps: 
     - uses: actions/checkout@v4 
     - run: pip install -r requirements.txt pytest 
     - run: pytest evals/test_evals.py 
       env: 
         OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} 
         JUDGE_MODEL: ${{ vars.JUDGE_MODEL }} 
 
 canary: 
   needs: eval-gate 
   runs-on: ubuntu-latest 
   steps: 
     - uses: actions/checkout@v4 
     - uses: google-github-actions/auth@v2 
       with: 
         workload_identity_provider: ${{ vars.GCP_WIF_PROVIDER }} 
         service_account: ${{ vars.GCP_DEPLOY_SA }} 
     - uses: google-github-actions/setup-gcloud@v2 
     - run: | 
         gcloud run deploy support-agent --source . --region=us-central1 \ 
           --set-secrets=OPENAI_API_KEY=openai-api-key:latest \ 
           --no-traffic --tag=canary 
         gcloud run services update-traffic support-agent \ 
           --region=us-central1 --to-tags=canary=10 

It uses short-lived workload identity instead of a JSON key and reads the provider key from a secret manager. The pattern carries over to ECS or Kubernetes. 

Step 6: Observe in production and roll back atomically 

Trace every run with OpenTelemetry or LangSmith: latency, token cost, tool-call success and escalation rate. Run online evals by scoring a sample of live conversations with your CI judge and rubric. 

Define rollback triggers: tool-call failures, cost spikes, safety-classifier flags, or judge scores below stable. Use sustained windows rather than single spikes, and start in alert-only mode. 

Roll back atomically. Revert the whole manifest, not just the image. With stateless containers, that’s a traffic shift back to the previous revision. 

Feed failures back. Every production failure becomes a new golden-dataset case, so the same mistake can’t ship twice. 

That’s a complete pipeline. For one agent, it’s enough. 

Going from one agent to production at scale? Get Lyzr’s playbook on how to take agents to production, covering governance, audit trails and deployment patterns for enterprise agents. 

What happens when one agent becomes fifty 

Six months later, the support agent has company: a sales agent in another framework, two agents on a different cloud, SaaS tools with built-in agents, and coding agents on developer laptops. Some have pipelines like this one. Many don’t. 

The pipeline still works, but it can’t solve these: 

• You can’t gate an agent you can’t see. Agents built outside the pipeline never hit your eval gate. 
• Every team’s gate is different. Thresholds, approvals and rollback rules drift across repos. 
• A YAML file isn’t an approval process. Who may promote to production, and where is that recorded? 
• Pipelines decide what ships, not what runs. Nothing stops a live agent’s next call when it exceeds budget or reaches for a disallowed tool. 
• Costs and rollout logic scatter across teams, providers and clouds. 

A pipeline ships an agent. It doesn’t govern it. At fleet scale, control has to live at the handoff into production and in the request path. 

Governing your pipeline’s output with Opencontroller 

Opencontroller is Lyzr’s control plane for AI agents across clouds, frameworks, models and runtimes. 

It doesn’t replace what you just built. Opencontroller doesn’t build or package containers; your team keeps owning agent code, images and deployment. Its architecture page puts it plainly: your CI keeps building, scanning and signing artifacts, while Opencontroller governs the handoff into production. 

It plugs into standards your pipeline already uses, including MCP, OpenTelemetry and OpenAI-compatible APIs. It runs inside your trust boundary, and customer-managed deployments don’t need to export telemetry or credentials to Lyzr. Basecode extends the same control to developer devices, and Lyzr says OpenController gives visibility into agent usage and spend across developers, projects and agents. 

The Friday incident, replayed. The eval gate fails refund-001. If a subtler version slips through, promotion needs an approval, the canary catches the drop on a small slice of sessions, and rollback is a pointer move. Meanwhile, request-path policy stops any agent in the fleet from overspending or calling disallowed tools. 

Conclusion: build the pipeline, then govern the fleet 

To build an agent deployment pipeline from scratch, treat the agent as more than its code: separate the core, pin every release in a manifest, gate on behavior, sandbox it, roll out gradually, and roll back atomically. That protects one agent. 

A fleet also needs a layer that sees every agent, applies the same release controls everywhere, and can stop a misbehaving agent in the request path. Keep the pipeline you built. Put a control plane on top of it. See how Opencontroller governs the handoff from your pipeline to production. 

FAQ

Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here