In 2026 almost every SaaS roadmap has an "agent" on it. Far fewer agents are in front of paying customers. Across the year's industry surveys, only a small minority of agent pilots reach production, and the blocker is rarely model quality. It is governance, reliability and cost.
This is the architecture we use when we take agents from demo to production for AI integration clients.
Start with the job, not the agent
An agent is a loop: the model reads context, picks a tool, observes the result and decides what to do next. That loop is powerful when the path is genuinely unpredictable. When it isn't, a plain workflow is cheaper and easier to debug.
Anthropic's widely cited guide, Building effective agents, makes the same point: start with the simplest pattern that works, such as prompt chaining or routing, and earn your way up to autonomy.
Write down three things before writing code:
- The outcome (for example, "refund approved or escalated within 2 minutes")
- The actions the agent may take, and which of them are irreversible
- The success metric you will put on a dashboard
Layer 1: tools designed for tasks, not endpoints
The most common mistake is giving the model your whole REST API. Forty endpoints with overlapping parameters create confusion, burn tokens and widen the attack surface.
Design tools around user intents: find_order, issue_refund(order_id, amount, reason), escalate_to_human. Each tool should:
- Validate inputs with a schema and return errors the model can act on
- Use scoped credentials for this tenant and this action only
- Log every call with the agent run ID
If you are exposing tools to external agents, the Model Context Protocol is now the standard way to do it.
Layer 2: an LLM gateway
Hard-coding one provider SDK is the 2026 equivalent of hard-coding one database host. Put a gateway between your code and the models for:
- Fallbacks when a provider degrades
- Per-tenant budgets and rate limits
- Prompt and response logging, with redaction
- Routing cheap tasks to small models
We cover the options in LLM gateways and model routing and the cost side in LLM API cost optimisation.
Layer 3: durable execution
Real agent tasks take minutes or days. They wait for webhooks, human approvals and rate-limited APIs. Running that inside a single HTTP request or a cron job is how you get half-finished refunds.
A durable workflow engine persists every step, retries safely and resumes after crashes. Funding for this category has grown through 2026 for exactly this reason. Our durable workflows guide explains the pattern.
Layer 4: memory you can audit
Memory is now a first-class feature in most agent frameworks. Treat it like any other data store:
- Short-term memory: the current run's messages and tool results
- Long-term memory: facts about the user or tenant, written explicitly and tagged with the source and time
- Retrieval: documents and records, with RAG evals
Every memory write should be inspectable and deletable. That is good engineering, and it is required for GDPR requests.
Layer 5: observability and evals
You cannot improve what you cannot replay. Trace every run: prompts, tool calls, latencies, token costs and the final outcome. OpenTelemetry's GenAI semantic conventions give you a vendor-neutral format.
Then build an eval suite from real traces:
- 50–200 representative tasks with expected outcomes
- Assertions on tool choice, arguments and final state, not just text similarity
- A run in CI on every prompt, tool or model change
Layer 6: security and approvals
Agents combine untrusted input with privileged actions, which is why prompt injection stays at the top of the OWASP LLM risk list and excessive agency sits close behind. The defences are architectural:
- Least-privilege tools, scoped per run
- Human approval for irreversible or high-value actions
- No direct path from untrusted content to data exfiltration
Our prompt injection defence guide goes deeper. If you serve EU users, check your AI Act Article 50 duties as well.
A realistic rollout
| Week | Milestone |
|---|---|
| 1 | Outcome, tools and eval set defined from real tickets |
| 2–3 | Agent in shadow mode: suggests actions, a human executes |
| 4 | Approved low-risk actions run autonomously, the rest need approval |
| 6+ | Expand autonomy as eval scores and incident data justify it |
Bottom line
Production agents are mostly ordinary engineering: good interfaces, durable state, observability and permissions. The model is the easy part.
Have an agent stuck in pilot? Talk to our AI team. We usually get a first production slice live within two sprints.
Frequently asked questions
Why do most AI agent pilots fail to reach production?
Usually not because of the model. The common blockers are governance (who approved this action?), reliability across long multi-step tasks, unclear cost, and security around tool access. Fixing those is an architecture problem, not a prompt problem.
Do we need a multi-agent framework?
Rarely at the start. A single agent with well-designed tools, a workflow engine for long tasks and good evals beats a swarm of agents that nobody can debug. Add more agents only when one clearly has too many responsibilities.
What is the minimum production stack for an AI agent?
Task-level tools with least-privilege credentials, an LLM gateway with fallbacks and budgets, durable execution for anything longer than a request, tracing of every step, an eval suite in CI and human approval for irreversible actions.