On this page
- A roleplay assessment scores a work conversation against a short list of criteria, each with a clear line between what counts and what doesn’t.
- AI scores the transcript criterion by criterion and quotes the lines behind each score, so a person can check every one.
- We check AI scores against expert assessors on the same sessions. They correlate at r = 0.80.
- Correlation is not agreement. Ask for the average gap between AI and experts, and for how well experts agree with each other.
What a roleplay assessment is
A roleplay assessment is a work conversation staged on purpose: one person plays their real role, someone else plays the other side, and an assessor scores what they do against criteria agreed in advance. The person playing the other side can be a trained actor, a colleague or an AI character. What makes it an assessment rather than practice is the scoring: the same situation, the same criteria and a score that can be compared across people and over time.
A simple example. A sales rep sends a proposal, and the buyer comes back with “It’s too expensive. The other vendor came in about 30% lower.” The rep has eight minutes to find out what is really behind the objection and leave with a next step. The assessor is not judging whether the rep is likable. They are checking specific behaviors: did the rep acknowledge the concern before defending the price, did they find the real constraint, did they agree on a dated next step.
That is the whole method. The rest of this article is about doing it well: writing criteria that two assessors read the same way, scoring at scale, and checking that the scores mean what you think.
Roleplay assessment in hiring vs in L&D
Most of what you find about roleplay exercises is written for candidates preparing for an assessment center. That is the hiring use. The international guidelines for assessment centers describe them as several components, including behavioral simulation exercises in which multiple trained assessors observe and record behavior1. The point is a fair, standardized decision about who gets the job or the promotion.
L&D teams use the same exercise for a different job: to see which skills a team already has and which ones need work. That changes what matters.
| In hiring | In L&D | |
|---|---|---|
| The question | Who should we select? | Which skills does each person need to build? |
| How often | Once per candidate | Repeated: a baseline, then again after training |
| What comes out | An overall rating and a decision | A score per skill, comparable across people and runs |
| After the exercise | The decision | A review of what was said and what to try next time |
Repetition is where L&D teams usually get stuck. A live assessor for every run of every person does not scale, so live roleplay assessments are hard to repeat. AI scoring removes that bottleneck. It is what lets you run a skills gap assessment on real conversations for a whole team, and repeat it, instead of relying on self-ratings. In hiring, the same idea shows up as structured first-round interviews scored by AI.
A roleplay assessment form, with an example
A good scoring form has a few criteria, and each one describes behavior you can point to in the conversation. “Good communication” fails that test. “Repeats the concern in her words” passes it.
Here is the form from one of our scenarios. The buyer is Nia · AI head of field operations at a regional HVAC services company. She likes the product, holds a cheaper competing quote and says the price is too high.
| Criterion | What counts | What doesn’t |
|---|---|---|
| Acknowledges before answering | Repeats the concern in her words and asks what the price is being compared to. | Defends the price in the first sentence. |
| Finds the real objection | Asks about budget, timing and the competing quote until the real constraint is on the table. | Treats the first objection as the only one. |
| Reframes on value | Ties the price to results from her own pilot and compares the two offers like for like. | Lists features instead of using her numbers. |
| Trades, doesn’t cave | Any change to price or terms comes with something in return: a longer term, a phased rollout, a reference call. | Offers a discount before she asks for one. |
| Agrees on a next step | Ends with a dated next step that includes the person who signs, such as a call with her CFO this week. | Closes with “I’ll send over a revised quote.” |
The situation, Nia’s profile and a full sample dialogue are on the price objection scenario with Nia. To build your own form, keep three rules from this one:
- Five criteria or fewer. More than that and assessors start scoring an overall impression.
- A “doesn’t count” for every criterion. The most common failure, written down, is what keeps two assessors consistent.
- Criteria from your own method. If your sales team works to a named methodology, or your managers to a competency model, the criteria should come from it, not from a generic list.
This article is about scoring the conversation. For the other half, getting reps ready for it, see how sales teams train on objections like Nia’s.
How AI scores the conversation
When an AI character plays the other side, the conversation is recorded and scored after every run. On The Reflection, scoring works in four steps.
- Transcript. The conversation, by voice or video, becomes a transcript with the time of every line.
- A score for each criterion. The AI scores the transcript against each criterion on the form, not as one overall impression.
- Evidence. Every score comes with the lines that earned or lost points, quoted with their timestamps. A reviewer can jump to that moment and judge it themselves.
- Scores by skill. Criterion scores roll up into the skills you track, so you can compare one run with the next, or one team with another.
What evidence looks like, for one criterion of an illustrative run. Owen is the rep.
5:52, Owen: “Let me put the numbers side by side and send them over.”
Lost points: no date, and the person who signs is not involved.
6:20, Owen: “Could we take your CFO through the comparison on Thursday?”
Earned points: a dated next step with the person who signs.
Skills that sound soft are scored the same way. Empathy, for example, is scored from what the person says: whether they name the other side’s concern and respond to it. The AI does not read the emotions, voice or face of the person being assessed. Only the AI character has a mood, as part of the scenario.
AI assessment validation: how we check scores against experts
An AI score is useful only if it agrees with the people whose judgment you already trust. So we test it the way you would test a new assessor: an expert assessor and the AI score the same sessions independently, skill by skill, and then the two sets of scores are compared.
Every client also has a module where their own internal experts can score the same sessions as the AI. That turns validation from a one-off study into something your team can watch: if the AI and your experts disagree on a criterion, you see it, and you can look at the quoted lines to find out why.
In our comparison, AI scores and expert scores correlate at r = 0.80. You can read more about the team behind the scoring method.
The most useful moment in a validation is the first disagreement. A team’s own experts score the same sessions as the AI, and on most criteria the two land close together. Then one criterion stands out. When we open the quoted lines, the cause is rarely the model. Usually two experts read the criterion differently too, because the “doesn’t count” line was missing or vague. We rewrite that line, everyone scores the next round against it, and the form gets sharper for people and AI alike. That is why we like teams to check the scores themselves: they end up with a better form, not only a number.
Why correlation is not enough
r = 0.80 means that AI and expert scores move together strongly: when experts score a session higher, the AI usually does too. It does not mean the AI is “80% accurate,” and it does not mean the two match in 80% of sessions. Correlation measures whether scores rise and fall together, not whether they land on the same number.
That difference matters. In 1986, the statisticians Martin Bland and Douglas Altman showed that correlation is the wrong test for whether two methods of measurement agree, and proposed looking at the differences between them instead2. A simple case shows why. If an AI scored every session exactly 10 points below the experts, the correlation would be perfect, and every score would still be wrong by 10 points.
So when you check any AI assessment, ours included, ask for three numbers, not one:
- Correlation. Do AI and experts rank sessions the same way?
- The average gap. Does the AI score systematically higher or lower than the experts, and by how much?
- Expert–expert agreement. How well do two experts agree with each other on the same sessions? Use it as a reference point when you interpret AI–expert agreement: it shows how far two qualified people differ on the same conversations. It is usually reported as an intraclass correlation coefficient (ICC), and the form of ICC and its confidence interval should be stated alongside it3.
When an L&D lead asks us how close the AI is to their experts, we ask a question back: how often do your two best assessors agree with each other? Most teams have never checked. So that is where we start. Two of your experts score the same set of sessions without comparing notes, and the AI scores them too. Now you have all three numbers from this section on your own conversations, and a fair benchmark: the AI is held to the standard your people actually meet, not to a perfect one. It is also an answer you can defend when leadership asks why they should trust the scores.
What a roleplay assessment can’t tell you
A well-built roleplay assessment is a strong signal, but it is still one kind of evidence. Three limits are worth stating to everyone who sees the scores.
- It is one conversation, not behavior in the field. A rep who handles Nia well has shown they can do it in a scenario. Whether they do it with real buyers on a busy Thursday is a separate question, best answered with what managers see and with results over time.
- Decisions about people stay with people. The AI produces scores and evidence. Who gets promoted, trained or moved is decided by your team, which can read the quoted lines and disagree with the score.
- The character is not a person. An AI character reacts consistently to what is said, which is what makes runs comparable. Real people are less predictable, so a good score is a reason to trust a skill more, not proof of it.
Questions L&D teams ask
How many roleplays make a reliable assessment?
More than one. A single run is a sample, and anyone can have a bad one. Assessment center guidelines rely on several simulations and more than one assessor for the same reason. For decisions that matter, use several scenarios that target the same skill and look at the pattern, not one score.
Who reviews the AI score?
Anyone you give access to. Every score comes with the quoted lines and timestamps behind it, so managers and L&D can check the reasoning, and your internal experts can score the same sessions in their own module to compare their judgment with the AI’s.
Can we score against our own competency model?
Yes. The criteria on a scenario can follow your own competency model or sales methodology instead of ours. Keep each criterion observable, with a clear line between what counts and what doesn’t.
Sources
- Deborah E. Rupp, Brian J. Hoffman, David Bischof et al. Guidelines and Ethical Considerations for Assessment Center Operations. Journal of Management, 41(4), 1244–1273, 2015.
- J. Martin Bland and Douglas G. Altman. Statistical Methods for Assessing Agreement Between Two Methods of Clinical Measurement. The Lancet, 1(8476), 307–310, 1986.
- Terry K. Koo and Mae Y. Li. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, 15(2), 155–163, 2016.
