AI support agents: how to measure ROI before and after launch

Every helpdesk now sells an AI agent, and every vendor quotes an impressive "automation rate". Six months later, many teams can't say whether their AI agent is saving money or quietly frustrating customers.

Here's how we measure AI support agents for clients, before and after launch.

Why deflection is the wrong metric

Deflection counts conversations that never reached a human. That includes customers who gave up, who couldn't find the "talk to a person" button, or who emailed instead. High deflection with falling satisfaction is a failure that looks like success on a dashboard.

Measure verified resolution instead.

The metrics that matter

Metric	Definition	Why
Resolution rate	Conversations solved by AI, confirmed by the customer or by no repeat contact in 72 hours	The real outcome
Cost per resolution	(AI platform + model + engineering costs) ÷ AI resolutions	Compare with human cost per ticket
CSAT (AI versus human)	Satisfaction on AI-handled versus human-handled conversations	Quality guardrail
Escalation quality	% of handoffs with full context, so the customer doesn't repeat themselves	Human efficiency and experience
Repeat contact rate	Same customer, same issue, within 7 days	Catches false resolutions
Human handle time	Average time agents spend per escalated ticket	AI should make the rest faster too
Wrong-answer rate	% of sampled AI answers that were incorrect	Risk and trust

Build the business case first

Before choosing a vendor, analyse 3–6 months of tickets:

1. Categorise by intent: order status, billing, how-to, bug report, refund and so on
2. For each intent, estimate whether it can be answered from docs alone, needs data lookup, or needs an action (refund, change plan, rebook)
3. Estimate volume × human cost per ticket × realistic AI resolution rate per intent

A simple example for a 10,000-ticket-a-month SaaS:

Intent group	Monthly tickets	Est. AI resolution	Tickets resolved
How-to (docs)	4,000	70%	2,800
Account and data lookups	3,000	50%	1,500
Actions (refunds, changes)	2,000	30%	600
Bugs and complex	1,000	5%	50
Total	10,000	~50%	4,950

Compare with the market benchmark. Intercom's Fin charges $0.99 per resolution, which gives you a useful reference for cost per resolution against your fully loaded human cost per ticket.

Buy versus build

Buy (Intercom Fin, Zendesk AI, Decagon, Sierra and others) when:

• You're on a mainstream helpdesk
• Most volume is how-to and policy questions answerable from docs
• You want to launch in weeks

Build or extend when:

• Resolution requires actions in your own systems through secure tools. See our agent architecture guide
• You need strict data residency or custom security controls
• Support is a core product experience, and you want control over tone, flows and cost

Many teams do both: a bought platform for the front line, plus custom tools (refund, rebook, account changes) exposed to it via API or MCP.

Launch safely

1. Fix the knowledge base first. AI amplifies bad docs. Retrieval quality decides answer quality; see RAG vs fine-tuning
2. Shadow mode: AI drafts answers, humans send them. Measure accuracy for 1–2 weeks
3. Limited rollout on the top three intents, with an obvious "talk to a human" option
4. Weekly review of a random sample of AI conversations and every complaint
5. Expand intents only when resolution and CSAT hold

Be transparent

Tell customers they're talking to an AI. It builds trust, and in the EU it's a legal requirement under AI Act Article 50.

Bottom line

Measure verified resolutions, cost per resolution and CSAT side by side. Ignore deflection. Build the business case from your real ticket mix, and extend bought tools with custom actions where the value is.

Our AI integration team runs ticket analysis, builds support agents and custom resolution tools, and sets up the measurement dashboards. If you're pricing your own AI agent, see how to price AI features. Book a call.