Chatbot owners want to know how well their bot is performing and often turn to containment as a north star to evaluate performance. But is it really? Let’s take a look at containment in detail to understand where it’s useful, and where it is lacking.
“Containment” is a metric that looks at how well a chatbot is automating conversations. That is, the ability to help customers get answers to their burning questions, without having to escalate to a live agent.
For businesses launching Conversational AI programs, it is often referred to as “deflection” or “case deflection”, a term that stems from the contact center, since it refers to the ability of a chatbot to divert incoming calls away from live agents, to automated services.
Containment is often considered a positive outcome, and as a chatbot owner, it is likely one of your key performance indicators (KPIs). Salesforce, in their annual State of Service report documents the growing trend for companies to see case deflection as a core KPI:
- 2018 36% of companies reported on it
- 2020 56% of companies
- 2022 67% of companies
Just last year, IT analyst firm Gartner estimated that “Chatbots Will Become a Primary Customer Service Channel Within Five Years (2027)” with containment identified as one area that must be improved, so companies are proud when they can announce containment rates in the range of 70%, 80% even 90% containment.
Unfortunately, while containment sounds like a noble goal, it doesn’t tell the full story of how well a chatbot is automating customer conversations.
Look beyond the north star of containment
While the business and chatbot creators might bask in the sight of a job well done through a high containment metric, the customer experience is not taken into account, and might in fact be altogether different. If your business is investing in automating the customer experience you need to understand what is happening behind the scenes of the containment metric.
Consider, for example, a few scenarios where the conversation is automated, yet the customer has an unfulfilling experience:
- Abandoning the conversation with the chatbot, without finding a useful resolution to their query.
- Getting caught in a loop of misunderstanding, before jumping ship to find help elsewhere.
- “Shouting” at the bot for help, which fails to escalate the customer because it doesn’t understand their plea.
- The customer finds a solution, but it might not be the optimal or right one.
Additionally, customers may complete a given intent, resulting in high containment rate, only to pick up the phone to engage a live agent because they were unable to get the answer they were looking for. This bypass is another way we lose the details on “true” containment, not to mention it doesn’t factor in the important detail of the experience.
How containment should be done
Automation should only be one of the values when measuring containment value. To understand “true containment” you need to consider additional factors:
- Does the user leave the chat conversation and escalate to a live agent, or did they abandon the conversation altogether?
- Was there any feedback when they exited the chat?
- Was there negative (or positive) sentiment expressed during the chat?
- Does the chatbot fail to reach a useful conclusion? (for example when a customer asks a question, and the bot is unable to interpret it accurately and so provides a misleading response)
- Do false positives lead the customer to unrelated responses?
Measuring containment in this way provides a more holistic, or nuanced view with better insight into whether the bot is performing or needs to be improved.
Measuring “True Containment” with chatbot analytics
Many chatbot platforms produce a value for containment rate as a key performance indicator. However, a chatbot analytics platform will give you a more accurate measure of how well the bot is automating the conversation.
Chatbot analytics will capture not only whether a conversation reached an intended end goal, but will also look at other signals found in the data escalation, abandonment, explicit feedback, AI-based sentiment analysis, false positive detection, and containment on topics that the bot was not intended to handle. It will score individual conversations and topics, and produce an overall score, but also allow you to see it on a more granular level.
At Wysdom, we refer to this as the Bot Automation Score, or BAS. With the BAS, product owners can fill in the gaps for a more complete picture of how well the bot drives customer experiences without escalation to an agent. You can learn more about BAS here.
As Conversational AI expands with more opportunities to engage your customers, being able to accurately assess containment is key to know what is happening under the hood and ensure that your customers are leaving with strong satisfaction.
