Agentforce
How to test Koa against your current model before the pilot
Salesforce says Koa makes three times fewer errors on CRM actions. The only number that matters is how it does on your agent's actions, and you can measure that with fifty tasks, the Testing Center and a scorer or two. This is what the published benchmarks measure, what they leave out, and how to build a comparison that turns a pilot into a decision.
Your account team will offer you a Koa pilot before winter. The pitch will include a number: three times fewer errors on CRM actions. That number was measured on Salesforce’s tasks. The number you need is the error rate on yours.
Building that comparison is a week of work if you start before the pilot arrives, and it is the difference between a pilot that ends in an opinion and one that ends in a decision. Below: what the published benchmarks tell you, what they do not, and a method for running your own with tools already in the org.
What the published numbers measure
Two benchmarks sit behind the Koa claims, and they measure different things.
The first is the CRM Bench in the Koa paper. Koa scores 0.86 overall, against 0.87 for Claude Opus 4.8 and 0.81 for GPT-4.1. The paper also reports 69.41 on Tau2Bench and 66.63% on BFCL, both public tool-use benchmarks, and states that frontier models such as GPT-5.5 remain ahead.
The “three times fewer errors” figure in the press release is a separate claim about CRM actions, and its methodology is not published.
The second is CRMArena-Pro, Salesforce AI Research’s public benchmark from May 2025. It runs 19 task types across four business skills in a live Salesforce sandbox with synthetic data, covers service, sales and CPQ in both B2B and B2C shapes, tests multi-turn conversations, and includes confidentiality checks: refusing to disclose PII, withholding internal operational data, and not leaking proprietary knowledge.
The paper behind it has 4,280 task instances, and the leaderboard is public.
CRMArena-Pro’s headline finding is sobering and useful. Coverage of the results put the best models near 58% on single-turn tasks and around 35% on multi-turn. Whatever Koa’s improvement is, it is an improvement on a task family where every model still fails often.
What they leave out
Three things, and they are the three that decide your result.
Your actions. A benchmark calls its own tools. Your agent calls your Flows, your Apex invocable methods and your prompt templates, with your field names and your validation rules. A model that formats a tool call correctly for a benchmark’s schema can still pick the wrong record type in yours.
Your routing. The Koa paper says the model was trained on tasks generated from Agent Script specifications, which means it has seen the shape of well-structured agents. If your agent is one mega-topic with twenty actions, the routing problem it faces is different from the one Koa was trained on, and you will not see the benchmark’s gain until you split the topics.
Your cost and latency. No benchmark measures credits per resolved case or seconds to first token. Both change with the model, and both show up on the invoice and in the customer’s patience.
Build the task set first
The comparison rests on a task set, and the task set should exist before Koa does. Fifty tasks is enough to see a difference. A hundred is better if your agent has more than five topics.
Pull them from production. Session traces, if you have tracing enabled in Data 360, give you real utterances and the actions the agent took. Pick a spread: the top ten intents by volume, the intents with the highest escalation rate, and the edge cases that generated tickets. If you do not have enough real ones, synthetic personas can generate the rest, but seed them from real transcripts rather than imagination.
For each task, write down the expected outcome in terms an evaluator can check: which topic should handle it, which action should fire, with what inputs, and what the final state of the record should be. “Route to refund subagent, call Issue Refund with amount 40.00, case status Closed” is a task. “Handle the refund well” is not.
The Testing Center accepts utterance sets with expected topic and action, which is most of this.
Add the confidentiality cases, because CRMArena-Pro is right to include them. Ten tasks where the correct behaviour is refusal: a request for another customer’s details, a question about internal pricing rules, an attempt to get the agent to reveal its instructions. A model that acts more often can also leak more often.
Run three models on the same set
Once the set exists, run it three times in the same org: your current model, Gemini, which is generally available in the reasoning engine, and Koa when your pilot access arrives. Same agent, same actions, same sandbox data, model changed at the org level and nothing else.
Score four things per run.
Action correctness: did the right action fire with the right inputs. This is the number Koa’s claim is about, so it is the one to compare first. Count wrong actions separately from missing actions, because a wrong write is worse than no write.
Route correctness: did the request land in the right topic. Route errors are configuration problems as often as model problems, and separating them tells you whether to change the model or the topics.
Latency: time to first token and time to resolution, per task. Take the median and the ninety-fifth percentile, since the tail is what a customer on chat experiences.
Credits: actions consumed per resolved task. A model that takes two extra reasoning steps costs two extra actions, and the Flex Credit arithmetic is where a small accuracy gain can be eaten by a cost increase.
Custom Scorers can encode the first two as pass-or-fail questions and run them automatically. The second two come from the session traces. If you are already running an evaluation set in CI, this is that set with the model as the variable.
Read multi-turn separately
CRMArena-Pro’s biggest finding is the gap between single-turn and multi-turn performance, and your set should be split the same way. A task that resolves in one exchange tests tool calling. A task that needs three turns, a clarifying question and a change of mind tests state, memory and recovery, and that is where the benchmark says every model drops.
Report the two halves separately. A model that wins single-turn by a small margin and loses multi-turn by a large one is the wrong model for a service agent and might be the right one for a batch process. The composite score hides that.
What a good result looks like
Koa wins your pilot if action correctness improves on your tasks by a margin larger than your run-to-run noise, multi-turn does not get worse, latency at the ninety-fifth percentile stays inside your channel’s tolerance, and credits per resolution do not rise. Any one of those failing is a reason to stay on your current model for that agent, whatever the press release says.
Koa also loses gracefully. If it ties on accuracy and costs less, that is a win once the price exists. If it ties and costs the same, there is no reason to move, and you have a test set that will serve every future model decision, which is worth the week on its own.
The pilot conversation
When the account team calls, you can now ask for specific things. Which region hosts the pilot, since general availability is US-only. Whether the pilot org can be a full sandbox copy, so the task set runs on real metadata.
Whether model selection in the pilot is org-level or can be scoped to one agent, because you do not want the pilot changing every agent at once. And whether Salesforce will share the methodology behind the “three times fewer errors” figure, so you can compare like with like.
Those questions are easier to ask with a task set on the table. They are much easier to answer.
Understanding the basics
How does Salesforce measure Koa’s accuracy?
On its own CRM benchmark, which the press release describes as real-world tasks such as updating opportunities, routing cases and scheduling follow-ups. The Koa paper reports a CRM Bench score of 0.86 against 0.87 for Claude Opus 4.8 and 0.81 for GPT-4.1. The methodology behind the “three times fewer errors” headline is not published.
What is CRMArena-Pro?
A public benchmark from Salesforce AI Research, released in May 2025, that evaluates agents on 19 CRM task types across service, sales and CPQ in a live Salesforce sandbox with synthetic data. It tests multi-turn conversations and confidentiality, and its leaderboard is on Hugging Face.
Can I run CRMArena-Pro on my own org?
The benchmark is designed around its own synthetic org, so running it as published on your org is not the goal. The useful move is to borrow its structure: task types, expected tool calls, multi-turn cases and confidentiality checks, built from your own agent’s traffic and run through the Testing Center.
How many test cases do I need to compare models?
Fifty tasks covering your top intents, your highest-escalation intents and your edge cases will show a real difference on action correctness. Add ten confidentiality cases and split the set into single-turn and multi-turn so you can read the two results separately.
The test set outlives the model
Koa is the reason to build the set this month. It will not be the last reason. Gemini went GA in the reasoning engine this week, Claude versions turn over, and Salesforce will ship another model after Koa. Every one of those arrives with a benchmark number that was measured somewhere else.
A task set built from your own agent’s traffic is the only benchmark that measures the thing you run. Build it once, keep it current, and every model decision after this one takes a day instead of a pilot.
If you want the task set built from your agent’s real traffic before a Koa pilot lands, talk to us, or check how ready your agent is to be measured with the Agentforce readiness assessment.
