How to Run a QA Calibration Session: A Practical Guide for BPO Teams

A practical calibration process for QA, Training and Operations teams: protect independent scoring, settle disagreements with evidence, and make sure every evaluator applies the same decisions afterward, including anyone who missed the session.

On paper, calibration is simple. Quality selects a call or interaction, everyone listens to it, everyone scores it on the QA form, and the team meets to compare.

In practice, it's rarely that simple. In more than 15 years in BPO, I've seen the same three problems come up again and again. Some evaluators who are short on time share their answers before the session. Not everyone makes it to the meeting. And evaluators can listen to the same call, use the same form and still disagree about the correct score.

This guide walks through a practical calibration process, stage by stage, with my recommendations for each of those problems built in.

Stage 1: Set the purpose and the standard

Before anyone listens to anything, every participant should know exactly what's being calibrated: the call or interaction under review, the version of the QA form that applies, and the definitions, rating anchors and approved policies that govern the scores. The form version matters most when the form has changed recently.

Those definitions and policies set the boundary for the whole session. Calibration aligns the team on a standard that already exists; it isn't the place to invent a new one. If the discussion shows that the form or a guideline doesn't cover what happened on the call, that's a finding to record and take to whoever owns the standard, not something to settle in the room and quietly start auditing against. I'll come back to this in stage 5.

When you choose what to calibrate, pick interactions that test the standard: a criterion that changed recently, an item the team keeps disputing, or a call that sits near the line between two ratings. If your criteria don't have written rating anchors yet, start there. How to Build a BPO QA Scorecard covers how to write them.

Stage 2: Plan the session and protect the time

Scheduling sounds like admin, but it's directly connected to the first problem. Calibration participants are busy. They have meetings, coaching sessions and operational tasks, and the calibration call competes with all of them. When people don't have enough time to listen to the interaction and complete the calibration sheet, some may share answers with colleagues beforehand.

So the planning has to make independent scoring realistic. Here's what I recommend:

• Agree on the calibration schedule with the participants and reserve the time in their calendars.
• Distribute the recording and the calibration form well before the session.
• Give people enough time to listen and submit their own assessment. Where it's practical, I recommend allowing a couple of days.

Confirm who needs to be there, too: the evaluators who audit that work, the person who makes the final call on disputed items (more on the master calibrator in stage 5), and any stakeholder whose expectations the scores need to reflect.

Stage 3: Make independent scoring non-negotiable

Sharing answers defeats the purpose of calibration. A participant can arrive with the correct score, copied from someone else, without understanding why that score was assigned or what evidence supports it. The team looks aligned, but nobody knows whether that evaluator would reach the same score alone. And alone is exactly how they'll audit the next call.

Copying also hides the differences you most need to see. If three people submit the same borrowed answer, a genuine misunderstanding in the group never comes to the surface.

I recommend two safeguards. First, remind participants explicitly that sharing answers before calibration is strictly prohibited. Put it in writing when you send the materials, so nobody treats it as a gray area.

Second, require evaluators to document the reason for each score as they complete the assessment, criterion by criterion. The reason should point to evidence: the exact wording, a timestamp, the behavior observed, or the criterion definition being applied. That makes the session more efficient, too. People explain their decisions from their notes instead of reconstructing their thinking from memory while everyone waits.

What you're aligning is the judgment behind the number: how the call is evaluated, how the evidence is interpreted, and which coaching opportunities get identified.

Pre-calibration checklist

Stage 4: Compare the results and find the real disagreements

The facilitator should collect every submission before the meeting and lay the scores side by side, criterion by criterion. That comparison tells you where to spend the session.

Look for two kinds of disagreement. The obvious kind is a criterion where the scores differ. The less obvious kind is a criterion where the scores match but the reasons don't. Those evaluators are likely to split on the next call.

Sample calibration agenda

Spend the meeting on the disputed items, replaying only the moments in question, and keep the discussion on the evidence, the definitions and the scoring rules. Watch for two shortcuts. One is the majority vote: if four people gave a 4 and one gave a 2, the four may simply have missed what the fifth heard. The other is everyone changing their score the moment the most senior person speaks.

Stage 5: Resolve differences with a master calibrator

Even with good preparation, evaluators won't always agree. Two careful people can read the same moment on a call differently. When the evidence discussion doesn't settle it, someone has to own the decision.

I recommend designating a master calibrator: one person with final internal authority over calibration decisions. That could be a senior QA evaluator, a QA supervisor or a QA manager, but seniority alone shouldn't decide who gets the role. The person should:

• be part of the Quality team;
• have a consistently high accuracy rate (I recommend at least 95% as a practical minimum when selecting a master calibrator);
• regularly liaise and calibrate with stakeholders or clients, and understand the client's expectations and the approved QA criteria;
• be able to explain scoring decisions with clear, verifiable evidence; and
• be willing to acknowledge an error and correct a decision when the evidence warrants it.

A word about that 95%. It's my benchmark from experience, not an industry requirement or a published standard, and it doesn't guarantee that every decision will be right. It isn't a 5% allowance for mistakes, either. A high accuracy rate shows that the person reliably applies the standard correctly. And because the rest of the team follows their decisions, the occasional mistake has to be acknowledged and corrected openly.

If your account calibrates with a client or other stakeholders, a small designated group can support that work, but internally everyone needs to know who owns the final decision. Once the master calibrator decides the score for a criterion, that decision guides everyone else unless it's reviewed and corrected through the process you've agreed.

Every decision needs its evidence

The master calibrator should always explain why a score was assigned, citing, where available:

• the exact words spoken on the call;
• the relevant timestamps;
• the specific action the agent took or missed;
• the definition or rating anchor being applied; and
• the approved guideline or procedure behind the decision.

That grounds the decision in facts rather than preference, and it gives the rest of the team reasoning they can reuse on the next call.

Example: settling a 2–2 split

When the master calibrator is wrong, or the rubric is

A master calibrator isn't infallible. When someone spots an obvious oversight, such as a moment on the call that was missed or a guideline that was misread, the professional response is to acknowledge it, correct the decision and make sure the correction is documented.

Genuine ambiguity is different. If the evidence or the rubric could reasonably support more than one reading, it shouldn't be settled by personal preference. Record the question and clarify it with the appropriate client or stakeholder. Gaps work the same way. In the example above, if the anchor had only said “confirms next steps,” the split would point to a gap in the rubric, not a mistake by two evaluators. Document the gap and get it clarified rather than creating a new rule and treating it as client-approved. If audits can't wait for the answer, agree on interim handling, label it clearly as pending confirmation, and revisit it when the clarification arrives.

A master calibrator's authority rests on competence, consistency, evidence, stakeholder alignment and accountability. The title alone doesn't earn it.

Stage 6: Document the outcome and share it with everyone

Attendance, the second problem I mentioned, needs its own plan. Not every participant makes it to the calibration session, and not every absence is negligence. Operational emergencies happen, and some conflicts can't be avoided. But an evaluator who missed the session still audits the next call, so they need the same guidance as everyone who was there.

Scheduling in advance and blocking calendars (stage 2) helps. Beyond that, I recommend:

• a clear expectation that participants attend, or let the facilitator know when they can't;
• comprehensive minutes from Quality whenever someone is unavoidably absent, documenting the agreed scores, the rationale behind them, the supporting evidence and the key takeaways; and
• sending those minutes to the relevant participants and stakeholders, so anyone who missed the session can understand the decisions and apply the same standard in their next audits.

An attendance record only tells you who was in the room. The minutes are what make sure every evaluator receives the same guidance and understands why each decision was made. That's why they need to show the reasoning, not just the results. An answer key tells people what to copy. A clear evidence trail teaches them how to evaluate.

Calibration minutes template

Stage 7: Follow through and check alignment over time

Calibration doesn't end when the minutes go out. The real test is whether the decisions show up in the audits that follow.

Check the next evaluations on the criteria you calibrated. If an evaluator is still scoring a disputed item the old way, follow up individually rather than waiting for the next session. Over time, keep track of:

• criteria that keep coming back as disputed;
• whether scoring agreement changes from one session to the next;
• the same misunderstanding reappearing, for one evaluator or across the team;
• rubric questions still waiting for clarification; and
• whether follow-up actions were actually completed.

No single number proves that calibration is working, so read these indicators together. When the same criterion keeps returning, look at the standard as well as the evaluators. The definition or anchor may need rewriting, which is a job for a scorecard review rather than another calibration round. Aligned evaluators are also the starting point for consistent coaching; Turning QA Scores Into Better Coaching covers that next step.

What good calibration gives you

Calibration won't eliminate every scoring error, and on its own it doesn't guarantee improvement. Run well, it gives a QA program audits that are consistent and defensible when they're challenged, fewer avoidable scoring conflicts, clearer expectations for frontline representatives, and better-targeted coaching.

When calibration works, evaluators do more than agree on a score. They learn how to recognize the right opportunities and coach the right behaviors.

Related guides

• How to Build a BPO QA Scorecard — Write the definitions and rating anchors that give calibration something concrete to apply.
• Customer Service QA Scorecard: What a 5 vs. 3 Looks Like — See rating anchors and evidence notes in a worked example.
• Turning QA Scores Into Better Coaching — Turn calibrated findings into specific coaching actions.
• Why I Built BPO Scorecard — The QA problems behind the tool.

Michelle is a BPO QA and training professional with more than 15 years of experience. Read more about Michelle and how these guides are prepared.