How to Maintain High Quality on Self-Service Channels

As more customer interactions shift to AI-driven and self-service channels, maintaining consistent quality has become increasingly complex.

So, how do you get it right when more traditional quality assurance (QA) methods don’t always translate well to automated environments? We asked the experts to find out…

Adapt Your Metrics and Reapply Them to Digital Interactions

Getting QA right for AI-driven and self-service channels means adapting the traditional metrics developed for human agents and reapplying them to digital interactions where no human is present:

• For Customer Effort Score (CES) – How easy was it to complete the task? How many steps were required?
• For First Contact Resolution (FCR) – How many of the AI-driven and self-service interactions subsequently required a call or email to a human agent?
• For Average Handling Time (AHT) – How long did they spend resolving their problem?

When automation is used in regulated industries, like healthcare or financial services, QA must also account for risk and compliance too. For example, did the chatbot say all of the compliance statements it was required to?

Ultimately, it’s about measuring outcomes, not just steps, and constantly refining the experience based on real-world results.

Contributed by: Martin Taylor, Co-Founder and Deputy CEO, Content Guru

Make Time to Test Off-Script and Edge Cases Too

Bots often perform well on ideal scenarios but fail on edge cases. Incorporate scenario-based QA testing, including off-script, emotional, or multi-intent queries. Use transcripts from real customer interactions to simulate these situations.

QA should also test how the AI handles frustration, ambiguity, or slang – especially in high-stakes situations like billing or cancellations. Include multilingual and accessibility QA as well.

A chatbot or IVR that works in a perfect environment might break down in the real world. Regular testing and retraining on real-world cases is the only way to ensure consistency and equity across all digital channels.

Contributed by: Tatiana Polyakova, COO, MiaRec

Set Benchmarks and Evaluation Criteria by Channel, Use Case, and Audience

Getting QA right on AI-driven and self-service channels requires a shift from rigid checklists to flexible, scalable frameworks.

What defines “good” in a chatbot interaction won’t look the same in email or voice – and your QA programme needs to reflect that.

It’s not just about measuring speed or resolution; it’s about understanding tone, intent, and user journey. Demographics also play a big role – what feels intuitive and effective for one customer group might frustrate another.

QA needs to consider this diversity, adapting benchmarks and evaluation criteria by channel, use case, and audience, rather than applying a one-size-fits-all model.

Contributed by: Tara Aldridge, Strategic Services Director, Vonage

Encourage Collaboration Between QA, Data Scientists, and Developers

Your QA team should continuously upskill to understand AI capabilities and limitations. Focus on how AI makes decisions, emphasizing transparency and ethics.

Encourage collaboration between QA, data scientists, and developers to ensure everyone is aligned too.

By investing in continuous training, you’ll ensure your AI and QA teams work together seamlessly to deliver top-notch service.

Contributed by: Khalil Rellin, Vice President of Operations at SupportZebra

Empower Customers to Vote on How Helpful Your FAQ Pages Are

Keep a close eye on how effectively your FAQ pages are performing. For example, by inviting customers to vote on how helpful your content is with a simple ‘thumbs up’ or ‘thumbs down’ scoring system.

Once this scoring system is in place, you also need to get into the discipline of regularly reviewing your answers and using those insights from customers to make sure you’re keeping your answers up to date and addressing any issues along the way.

Contributed by: Vinay Parmar, Managing Director at Customer Whisperers Limited

Make Sure Your QA Process Accurately Reflects the Strengths and Limitations of AI

Traditional QA scorecards often fall apart when applied to AI-driven and self-service channels. Why?

Because AI doesn’t build rapport, express empathy, or improvise – it executes based on training and data. Scoring a virtual agent on human criteria like tone or soft skills misses the point.

To get QA right in automated environments, you need rubrics designed for bots. Prioritize things like:

• Was the customer’s intent correctly understood?
• Was the answer factually accurate and complete?
• Did the hand-off to a human happen at the right moment?
• Was the latency acceptable?

Your QA should reflect the strengths and limitations of AI – and be built in collaboration with product and engineering.

These teams can help identify key failure modes (like looping, hallucinations, or poor routing) and ensure QA becomes part of a tighter feedback loop. It’s not about checking if the bot was “nice”. It’s about whether it was right.

Contributed by: Briana Tischner, Product Marketing Manager, Assembled

Capture Where Channels Are Performing Well AND Where They Are Causing Customer Frustrations

The majority of contact centres do not have the level of insight into their customer interactions that they should.

This only worsens when it comes to AI-driven and self-service channels, where there can often be no oversight into the quality of service being provided.

But analytics tools can change that, as they are fully integrated into conversation intelligence solutions to provide a comprehensive view of operations, and automatically evaluate 100% of interactions – from phone calls to chatbots, live chats and emails.

They rate interactions on a range of KPIs, including those based on sentiment and tone, such as empathy and professionalism.

This enables contact centre managers to quickly and easily gain insight into where these channels are performing well, and where they are causing customer frustrations.

Equipped with these insights, managers can take action to make improvements, such as tuning the AI or updating FAQs, to ensure automation and self-service works as intended, no matter which channel is being used.

Contributed by: Magnus Geverts, VP Product Marketing, Calabrio

Tweak and Improve Performance as It Happens – Not After the Fact

As more interactions move to AI and self-service, supervisors can’t rely on static QA reporting. Real-time dashboards and feedback loops are key to keeping quality high.

Tracking things like sentiment, drop-off points, and resolution rates gives you the insight to tweak and improve performance as it happens – not after the fact.

It’s also important to bring QA into the early stages of flow design and keep reviewing regularly.

The goal isn’t just to catch mistakes; it’s to build a system that learns and adapts. In this space, quality management needs to be proactive, flexible, and constantly evolving.

Contributed by: Carl Townley-Taylor, Product Manager, Enghouse Interactive

Use Your Data to Uncover Repeat Contacts, Channel Hopping, and Dead Ends

These signals often reveal where automation is falling short and experience quality might dip.

QA measures should flag gaps before they damage relationships.

Don’t Ignore Human Oversight

Automated QA tools can highlight trends, but real improvement comes from your people who understand the bigger picture.

Keep humans in the loop to review performance, tweak business processes and drive smarter service design.

Contributed by: Lewis Gallagher, Senior Solutions Consultant, Netcall

Stop Reviewing AI and Self-Service Channels in Isolation

Quality isn’t just about compliance and ticking things off a list any more. It’s about how your customer feels in that interaction, and so QA needs to embrace a wider scope of what quality is.

From the customer’s perspective, this means building in QA frameworks that really evaluate your customer’s emotional outcome in their experience.

After all, if we’re going digital and a significant proportion of our interactions with customers are going to be via AI and self-service, we really need to be thinking about key touchpoints and asking ourselves, “How do we create that same connection with the brand?”

It’s about looking at the customer journey as a whole – not just reviewing AI and self-service experiences in isolation.

Contributed by: Katie Stabler, CULTIVATE Customer Experience by Design and author of CX-Ism: Re-Defining Business Success

Detect Where Human Takeover Should Have Occurred – But Didn’t

Manual QA is already resource-intensive for voice channels – it’s virtually impossible for self-service and chatbots without automation.

AI-powered QA tools can evaluate 100% of AI-driven interactions. These systems can automatically assess chatbot accuracy, misroutes, sentiment shifts, and customer frustration cues.

They can also detect where human takeover should have occurred but didn’t. Using AI to QA other AI might sound ironic, but it’s the only scalable path forward.

Bonus: AI tools can flag areas where automation needs refinement or where the bot hands off too early/too late, turning QA into a feedback loop for continuous improvement.

Contributed by: Tatiana Polyakova, COO, MiaRec

Don’t Just Look at How Well Your Self-Service Options Are Working, Consider Why Customers Need to Use Them in the First Place

It doesn’t stop there! You also need to consider what insights you can deliver back to the organization about your end-to-end customer experience.

It’s about addressing friction points at their source, because if you reduce quality to whether “I answered the query” and “the customer liked me”, I think you’re missing 50% of the equation.

Yes, you should be taking care of the basics and resolving the query in the moment, but the question in your mind should always be “What made that customer have to make contact in the first place?” and seeing what you can also do to address the root cause – so those queries don’t come into the contact centre via any channel.

Contributed by: Pier Ragone, Principal Consultant, CX Operations & Strategy

For more great insights and advice from our panel of experts, read these articles next:

• Have Wallboards Had Their Day?
• What Are Intelligent Contact Centres Doing Right Now?
• 15 Proven Tactics to Reduce Abandon Rate
• Top Tips for Digital Channels – Forecasting and Scheduling

Author: Megan Jones
Reviewed by: Jo Robinson