How to Tell If Your Intercom Fin AI Agent Is Actually Working

# How to Tell If Your Intercom Fin AI Agent Is Actually Working

Most businesses turn Fin on, watch the automation percentage tick up on the dashboard, and assume the job is done. That number alone will not tell you what you need to know. A high automation rate can hide a bot that is quietly frustrating customers, and a modest one can belong to a setup that is working exactly as intended for a complicated product. If you want a real answer to whether this thing is paying for itself, you have to look at more than one line on a chart.

Start With the Metrics Intercom Actually Gives You

Intercom's support performance view breaks Fin's activity into a few distinct numbers, and they answer different questions.

Automation rate tells you what share of all new conversations Fin resolved on its own. It is the headline number, and it is also the easiest one to misread, because it blends together conversations Fin never should have touched with ones it fumbled.

Involvement rate tells you how often Fin actually stepped into a conversation and tried to help, out of everything that came in. Some volume never reaches Fin at all, whether because of routing rules, business hours logic, or channels Fin isn't set up on.

Resolution rate is the more honest number for judging Fin's actual skill. It measures, out of the conversations Fin got involved in, how many it resolved without a human stepping in. This is where you catch the gap between Fin simply showing up in a conversation and Fin actually solving the problem.

There is also a customer experience score built from ratings on conversations Fin handled by itself, factoring in things like whether the answer was accurate, how the customer seemed to feel, and how much effort the exchange took. This is the number that catches the cases where Fin technically closed a ticket but left the customer annoyed on the way out.

Why Watching Automation Rate Alone Backfires

If you only track automation rate, you can talk yourself into two very different wrong conclusions. A rising automation rate with a falling CX score usually means Fin is closing conversations too aggressively, marking things resolved that customers didn't actually consider settled. A flat automation rate with a healthy CX score can mean the setup is fine and the ceiling is just lower for your type of support volume, maybe because a large share of your tickets genuinely need a human (billing disputes, account-specific troubleshooting, anything requiring judgment calls).

The fix is to look at involvement rate and resolution rate side by side. If involvement is high but resolution is low, Fin is getting pulled into conversations it can't finish, which usually points to gaps in your help center content or workflow logic rather than a fundamental limitation of the AI.

Let CSAT Catch What the Percentages Miss

Fin can trigger a satisfaction survey after it gives an answer, when a customer goes quiet, or both, and you choose which condition fires it. The resulting CSAT report shows your positive rating percentage along with a sentiment breakdown, and on higher-tier plans you can pull the underlying rating remarks for detail.

The setup detail that actually matters here: if you don't separate AI-only conversations from ones a teammate also touched, your CSAT number becomes a blend of two very different experiences and stops telling you anything about Fin specifically. Building a workflow branch that filters out conversations where a teammate replied, before the survey fires, keeps the data clean enough to act on. It's also worth reading the written remarks rather than just watching the percentage move, since a customer who leaves a middling score with a specific complaint is more useful to you than ten quiet 5-star ratings.

Know What You're Actually Being Billed For

Fin's outcome-based pricing means you're charged for specific outcomes, not raw usage: a resolution, a procedure handoff to a human or workflow, and for Fin for Sales, a qualification or disqualification. Only one outcome gets billed per conversation no matter how many messages or actions happened inside it. Escalations that happen because of your own workspace rules, a failed procedure, spam filtering, or a customer who simply stopped responding after a clarifying question don't count as billable outcomes at all.

That last point matters for anyone reviewing costs against results. A spike in unbilled escalations isn't necessarily Fin failing, it might be your rules routing more conversations to humans than needed, or it could be customers abandoning conversations before Fin got a real shot at helping. Both are fixable, but they're fixed differently, and mixing them together on a spreadsheet leads to the wrong conclusion about where the problem is.

Make This a Monthly Habit, Not a One-Time Setup Check

The businesses that get the most value out of Fin treat this as ongoing tuning, not a launch-and-forget project. A simple monthly routine works well: check resolution rate against involvement rate for drift, skim CX score for dips tied to specific topics or workflows, read a sample of CSAT remarks instead of just the score, and pull the escalation breakdown to see whether handoffs are trending toward user-requested (a content or scope gap) or rules-based (a workflow configuration issue). Intercom's own optimization tools will usually flag specific weak spots once you know which metric to go looking for.

None of this requires guesswork. The data is there, it's just spread across a few different views, and it's easy to stop at the first number the dashboard puts in front of you.

If you'd rather not spend your afternoons reverse-engineering your own Fin metrics, this is the kind of tuning work I do for businesses running Intercom day to day. Feel free to reach out if you want a second set of eyes on your setup.

*Cover photo by Carlos Muza on Unsplash.*