Every chatbot owner wants to learn how to improve bot performance. The first place they’ll turn is intent analysis, to figure out what’s working and what’s not.
In our recent webinar, Jay Athia, Senior Director, Customer Success walks through a demo of the Operations Center, giving a new lens to think about how to measure intent performance.
Here are his top 3 takeaways:
1. Analyzing intent performance isn’t enough
Intent is based on how the business thinks about the conversation journey. But, that’s not the same as what the customer is thinking about.
To have the most impact, a bot improvement plan needs to take into consideration real-world customer conversations.
Think of contact reason as an audit for your intent – when you inspect contact reason you are able to identify gaps between what’s been programmed (ie, the intent) and what’s actually happening.
2. Containment is a false measure of bot performance
It’s easy to think that containment tells you how well your bot is automating conversations. However, it’s a false signal that more often than not, actually masks the problem.
The reality is that containment only tells you what happened within the bot itself. It does not provide insight whether the customer exited the bot to a more expensive channel. Other signals that you should be looking for include automation rates and sentiment – both positive and negative sentiment.
Even more importantly, you should be looking at live agent call handle time for the same topic. By doing so, you’ll get the full picture of intent performance.
3. Signs to look for when planning your improvement roadmap
When planning your bot improvement roadmap, you’ll probably have a lot of ideas. You’ll probably even get input from others across the business about what you should focus on next.
But unless you are looking at performance from a few angles, your changes won’t have the impact you’re hoping for.
- If you look at performance according to a specific topic, you may notice that containment is falling, so the tendency would be to fix it. But in fact, looking at experience metrics you’ll see that the bot is actually performing quite well.
- On the flip side, looking only at containment rates, it might seem that you’re doing well, when in fact, digging deeper into automation or sentiment, you’ll see that customer experience is actually falling.
Measuring performance across a suite of metrics gives you a more holistic understanding of what’s working or not, so you can make sure your roadmap is addressing functionality that will have a positive impact.
There’s no one-size fits all approach to analyzing intent performance. But, with the right framework in place, you’ll be in a better position to identify what’s actually working (or not) and build the right bot improvement plan.
