How to Measure Chatbot Performance

---

16 Practical Ways to Measure Chatbot Performance

You’ll never know how well your chatbot is truly serving your customers if you don’t measure this accurately in your contact centre.

Not sure where to start? Here are 16 expert ways to measure the performance of your chatbot:

1. Draw Comparisons With Other Channels

Start with measuring success in terms of your initial objective and any comparisons that can be drawn with other channels.

Consider looking at the number of cases handled, the time spent with the chatbot, and any reduction in handling time when these cases are escalated rather than going directly through an agent-led channel.

Then consider net promoter score, reuse rates and first contact resolution, These can predict long-term adoption and therefore whether your economic performance is sustainable.

2. Utilize Mystery Shoppers

To check accuracy, intent recognition, response quality, errors and contextual understanding, a numerical score doesn’t provide the details needed to make impactful changes.

Therefore, one way to assess chatbot performance is to have an independent party run through scenarios and questions and report on what they find. This approach helps identify any problems that may be encountered when callers deviate from the script.

Why? If the same people who set up the chatbot run the practice questions, they will use the same language they used to train it, most likely leading to a positive performance bias.

Contributed by: Grace Dawson at Odigo

3. Use a Feedback Loop to Uncover Ineffective Chatbots

Chatbots often fall short of customer expectations by failing to comprehend requests or provide satisfactory resolutions.

Advanced AI solutions can bridge this gap by linking unattended chatbot conversations with attended human-agent interactions.

This creates a feedback loop that analyses both types of interactions to uncover ineffective chatbots. The AI solution then uses insights from the highest-performing human agents to train the chatbots on how to properly resolve similar issues.

Businesses can significantly improve virtual assistant performance by continuously optimizing chatbots based on real conversational data. This empowers brands to increase self-service containment rates, seamlessly elevate complex issues, and deliver satisfying customer experiences across digital channels.

Contributed by: Andy Traba at NICE

4. Analyse the Chatbot Conversations

One way of learning what is and isn’t working is analysing the chatbot conversations – just as you might with conversations between human agents and customers – to uncover insights and identify common patterns or issues that are related to chatbot failures.

For example:

• What are the most common questions or issues that chatbots can’t solve?
• Is there an increase in customers mentioning chatbots once they do get to an agent?
• If so, is that sentiment positive or negative?

Using these insights can help you better understand why your chatbot isn’t performing as it should.

By identifying the root cause, it’s then easier to make improvements, such as tweaking the algorithm or increasing training data, so customers can experience a better, more streamlined service journey.

Contributed by: Frank Sherlock at CallMiner

5. Evidence Tangible Time Savings for Agents

Calabrio research suggests that 40% of agents welcome innovative AI-powered tools like chatbots to free them from tedious, routine tasks so they can focus on more fulfilling, higher-value activities.

Contact centres receive countless routine interactions every day, so if you can automate as many as possible without affecting service levels, you will reap significant time savings for agents.

Try out the following formula for an accurate calculation:

Savings = (number of correctly answered questions/agent’s average response time/ agent’s average rate per hour) minus (bot creation cost + service subscription cost).

If the value is positive, the chatbot can be scaled up or extended to other channels. If the value is negative, consider increasing the number of questions that the chatbot answers and check the correctness of the answers.

6. Measure Total Interactions vs New Interactions

Chatbot success is all about customer re-engagement, so if people are returning to your bot for a variety of queries, this suggests they are happy with the service. Simply divide your total number of chatbot users by the number of new chatbot users to establish a baseline.

If the total number of users seems lower than expected, there may be a technical problem with your chatbot that prevents customers from using it. If the number of new users is decreasing, you may need to reconfigure the chatbot. Is your chatbot answering the right questions?

Website FAQs are a good place to start – providing they are written in the customer’s language. Consider deploying modern speech analytics to identify common questions asked by customers.

This exercise is also a powerful brand indicator. If you’re using a chatbot alongside a marketing campaign, new user spikes will generally indicate high levels of interest and engagement in the campaign.

7. Keep Tabs on CSAT Scores

Customer Satisfaction (CSAT) is a metric that applies to any service, and monitoring CSAT for your chatbot is no different from monitoring your agents.

When CSAT is much higher for your customer service team than your chatbot, the bot is probably not performing to customer expectations. If satisfaction with the chatbot is significantly higher, then there might be areas of improvement in your contact centre.

You should aim for similar numbers across agent and chatbot scores. If the CSAT is acceptable, this means that both units are doing their job.

To find out how satisfied customers are with your chatbot, build effective feedback loops into the equation. A quick and easy solution is to add questions about the chatbot into your current CSAT survey. For instant feedback, include a message at the end of a customer’s interaction with the chatbot, asking them to give a thumbs-up or down or even a 1–5 star-rating.

Contributed by: Florian Garnier at Calabrio

8. Monitor Abandonment Rates

One of the metrics to prioritize is abandonment rate. According to our recent data, customers leveraging AI-powered virtual agents experienced significantly fewer abandoned calls than those without chatbot technology.

Lower abandonments rates will show that your chatbots are able to provide quick answers to easy questions and quickly route customers to a human agent when interactions become complex.

After all, recent studies show that 67% of consumers prefer self-serving than speaking to a customer service representative.

Contributed by: Jason Griffin at Five9

9. Avoid Using a Single Metric

Tracking the right key performance indicators (KPIs) is of utmost importance when measuring the success of chatbots. It is imperative to avoid the pitfall of designing and building chatbots based on a single metric, such as containment rate, which can lead to skewed outcomes.

Instead, 360° analytics will help ensure that teams are capturing actionable data, content, and outcomes from every customer chatbot interaction, providing end-to-end visibility into contact centre operations.

Having a holistic view of quality, actionable metrics is also necessary for measuring success. Factors like conversion rate, true automation and customer experience should all be considered when evaluating the quality of bot interactions.

Real-time insights will allow benchmarking against service level agreements (SLAs) and critical KPIs, ensuring service quality remains consistent even during surges in contact volumes, such as during an outage or emergency.

Furthermore, aggregated insights provide valuable trends from past interactions, enhancing forecasting and contact centre planning. By drilling down into specific customer journey paths, friction points can be identified and customer experiences optimized.

Contributed by: Jonathan Mckenzie at 8×8

10. Implement Sentiment Analysis to Vibe-Check Interactions

Ever been stuck in chatbot hell – that infuriating cycle of repetitive replies that leaves you typing REAL AGENT NOW in all caps? Sentiment analysis can help you ensure your customers never have to go there.

Using a combination of machine learning and natural language processing, sentiment analysis allows you to determine the vibe of a user’s experience by tracking the emotional ebb and flow of their chatbot journey.

These insights can illuminate the kinds of responses and interactions that push a customer’s frustration button, as well as those that appear to facilitate intuitive and hassle-free experiences.

Contributed by: Kaytlin Louton at Assembled

11. Compare Answers to Predefined Scripts

Over the years chatbots have become a crucial interaction channel in the customer communications mix. But like any other channel, you need to make sure you are gauging its effectiveness and measuring its performance.

The key to measuring chatbot performance lies in evaluating its ability to deliver precise and pertinent responses. This can be done by comparing the answers against predefined scripts to gauge accuracy.

12. Look at What Can Be Done WITHOUT Human Intervention

Take a look at the completion rate, i.e. the percentage of customer interactions the chatbot handles successfully without requiring human intervention.

A high completion rate indicates the chatbot’s self-sufficiency and ability to handle a wide range of customer enquiries independently. Not only does this lighten the load for agents, but it also improves first-contact resolution and overall customer satisfaction.

13. Analyse the Chatbot’s Response Times

For your chatbot to be effective you need to ensure that you are continually optimizing its performance. To do this several strategies come into play, including analysing the chatbot’s response times against predefined targets. This helps ensure you are providing timely assistance.

Contributed by: Colin Mann at Enghouse Interactive

14. Check Your Chatbot Escalation Rate

When a high percentage of interactions require escalation or you are experiencing low containment rates, something is clearly amiss. However, that isn’t a conclusion in itself, there are still several things that could be going wrong, such as:

• Is it that the chatbot isn’t accurately recognizing intent?
• Is the quality of the answer poor?
• Or is it simply that chatbots aren’t a great solution for the context of the query in question?

Check your other metrics (such as CSAT or NPS) for customers who don’t escalate and how the chatbot’s answers compare to an ideal agent-generated response? Sometimes there is a query subset that could be diverted at an earlier stage through multiple-choice options for more specialized support.

15. Evaluate for Common Pain Points

Identifying an underperforming chatbot is not the same as knowing how to improve it.

For that, you need to actually diagnose the problem using a range of KPIs and evaluate for these common pain points:

Contributed by: Grace Dawson at Odigo

16. Ask Phase One and Phase Two Questions

Too many customers and companies deploy chatbots and do not take into account the online experience at the time.

It is really important a chatbot is launched at the right time. For example, if a cart looks like it is about to be abandoned, this is the time to launch the chatbot, not just when someone lands on the page, as this becomes a dumb chatbot.

Design into a chatbot the ability to escalate the interaction to an agent if it is needed; it cannot be a chatbot-only experience, especially if it ends with the message, “sorry, I cannot help you. Ring this number.”

When considering feedback from customers, there are two areas a survey or feedback needs to cover:

1. What has your online experience been?
2. What has your overall experience been?

Phase one and phase two questions

Currently, they only ask what your online experience was like, but this doesn’t give you an overall understanding of how the chatbot is doing.

Contributed by: Nick Wingrove at Genesys

For more great insights and advice from our panel of experts, read these articles next:

• How to Motivate Staff and Drive Employee Engagement
• 14 Best Practices to Streamline Your Incoming Customer Queries
• How to Improve Contact Centre Response Time
• Contact Centre Performance Challenges and How to Fix Them