Join us for a live demo of the Operations Center on December 6th at 11:00 AM ET  – Register Here

BLOG

The top 3 enterprise bots: Performance metrics that actually matter

Share this 

Share on facebook
Share on twitter
Share on linkedin

Recent analyst studies state that one in ten customer interactions will be automated by 2026, representing a dramatic increase from the estimated 1.6% of interactions today. As we pivot to more digital engagement, organizations need to prepare for a chatbot strategy that includes a plan to measure and improve performance in order to meet internal automation goals while offering your customers the same high-touch experience they have come to expect.

The 3 big enterprise chatbot

Enterprise chatbots typically fall into one of three categories 

  1. Customer service bots – to help you take care of your existing customers
  2. Marketing bots – to help you acquire new customers
  3. Employee service bots – to help you take care of your team

When managed well, they provide your organization with a competitive edge. But if poorly managed, they quickly turn into a liability that damages customer experience, hurts your brand, your NPS or CSAT scores, and in the extreme, has a negative impact on your bottom line. 

In this blog, we will go over each of these bots, how they can help your business, and how to know if they’re delivering what they should. Most importantly, we will show you which KPIs you should be tracking and what it takes to improve them continuously.  

Chatbots for customer service

Customer service bots are the oldest and most deployed of the big 3 bots and they can have a dramatic effect on the performance of your contact center. The primary goal of this chatbot is to deliver high-quality service to your existing customers, at the best possible cost, and avoid escalation to a live agent in your contact center. 

The data you’re going to want to focus on here are detailed containment metrics.  

How to measure service bot performance

The overall containment number that most bot vendors publish doesn’t tell you the full story. What you need is a detailed analysis that takes into account good containment vs. bad types of containment. We consider this “real containment”.

In order to measure “real containment” rate you need to consider the number of times your customer 

  1. Engaged about topics that were assigned as a bot topic, in order to identify which topics your bot will handle vs. the topics live agents will handle.
  2. Reached a bot endpoint, which can be identified when they are tracked as events within your bot platform.
  3. Requested a live agent, versus those who didn’t

You then need to remove from this analysis: 

  1. False positives. Surprisingly, many companies have a rate of 20-40% false positives, and don’t even realize it. 
  2. Abandonment while waiting for an agent, especially when it has been escalated by the system. Many customers will not wait more than a few seconds for a live agent.
  3. Bad containment. These are the topics that the bot was not designed to contain and should have gone directly to a live agent, but your customer was mistakenly routed into a bot journey. 
  4. Customers who leave negative bot feedback. Despite your best efforts, sometimes customers are left unsatisfied even when all other criteria have been met.

By evaluating all of these criteria together, you’ll achieve a much stronger summary of your “real containment”, giving you the insight needed to take appropriate next action.

Chatbots for marketing

Marketing bots are all about generating qualified leads for your sales team, keeping them engaged and curious, ready to convert to a purchase decision. Having a marketing bot working 24/7 qualifying leads and intelligently routing them ensures your prospects are never waiting, so you can maximize your lead generation investments. 

How to measure marketing bot performance

You will want to measure all of the typical bot metrics like CSAT, routing efficiency, and rate of natural language understanding (NLU), but you will also want to zero in on contact reasons. Determining why a prospect wanted to engage is key to the success of a marketing bot, and requires a distinct approach called topic modeling.

A topic model is an AI model trained using the data from millions of chat conversations between the prospect and the bot, and prospect and and live agent, to identify the true reasons for contact. So, whereas your conversation journeys may initially be designed around what you think your prospect wants to know, a topic model will tell what topics you might be missing. 

A well-trained topic model will let you continuously optimize your lead generation efforts based on exactly what your prospects want to talk about every day, and ensure you’re identifying those additional prospects for your sales team.

Chatbots for employee service

Internal chatbots are a somewhat newer variation of the customer service chatbot.  These started as IT helpdesk bots and have evolved to include anything that your employees may need to ask the business about. Typical use cases include:

  • HR information
  • Recruiting
  • Onboarding
  • Training
  • Document search
  • IT helpdesk

These bots are usually made available through internal messaging tools such as Slack, Microsoft Teams, and Google Chat. 

The key to measuring the effectiveness of these bots is similar to the customer service bot where you’ll want to measure “real containment”, while also keeping a close eye on employee satisfaction.

How to measure employee service bot performance

To measure ESAT from an internal bot you need to go beyond the typical thumbs up/down or freeform feedback fields. A more holistic model should consider the following: 

  1. Abandonment rate: How often does the employee exit the chat before reaching the intended endpoint.
  2. Paraphrase rate: How often the employee says the same thing within a given session.
  3. Ditto rate: How often the bot gives the same answer multiple times or more within a session.
  4. Conversation sentiment: These models don’t always get it right but are helpful as input within a model.
  5. Profanity rate: You’d be surprised at how often people curse at bots!
  6. Number of handoffs, post escalation: If escalation to live agents is available, then what number of transfers between agents are needed to resolve the issue.
  7. Feedback: Freeform text analysis provides a wealth of information that should be considered in the analysis. 

A single view for all of your chatbot performance

The most important thing to consider as you roll out these bots is that you should be managing the big 3 (and more) in one place. Bringing together all your different bot types, regardless of platform or channel, under one consolidated view will give you a clearer picture of where your customers, prospects, and employees are getting value, insights into the real topics they’re engaging with your bots, and where you need to still improve (and bots can always get better). 

Getting this right will mean that your team will now have the ability to quickly spot what needs improvement with a clear path to improve performance, meaning they will spend less time dealing with administrative challenges, and more time getting real work done.

How artificial intelligence is advancing the performance of chatbots 

Chatbots are a great asset to organizations, and despite tremendous advancement in the past few years, yet they are still in their infancy compared to what’s coming. Every month that goes by, new technologies and innovations are introduced, so you can expect that your bots will quickly evolve from their current state. Artificial intelligence, for example, offers new opportunities for bot leaders. LIghtning fast text classification and topic clustering, natural language understanding, and sentiment analysis, are all examples of how AI is serving to advance bot technology.

The most important thing to remember is that the performance of your bot is only as good as the improvements you can make. Knowing what to measure is the first step to put you on the path to success, so you’re ready when the next wave of bot innovation presents itself. 

Want more Wysdom? 

Learn about the 6 key metrics every chatbot team should be measuring on any chatbot platform in our article Wysdom’s 2022 State of the Chatbot Industry.

We use cookies to ensure that we give you the best experience on our website. By clicking “I Accept” or if you continue to this site, we will assume that you consent to the use of cookies unless you have disabled them.

Data Scientist

As a Data Scientist, your day-to-day will involve writing queries, building dashboards, and preparing analytical reports about product performance for our clients and the Wysdom.AI team.  You will work with SQL, Tableau, and Python and ML Frameworks/libraries (among other tools) to create stunning visuals showing how Wysdom.AI is making their customers’ experience even better.

Implementation Engineer

As an Implementation Engineer you will be responsible for the solution integration of an enterprise grade conversational AI experience from a technical perspective. You will work closely with the lead solutions architect and be the technical face of the implementation team and lead the customer through the entire implementation cycle. You will work with one of the most diverse teams of linguists, data scientists, and innovators to deliver the best AI enabled customer experience.

Solution Architect

As a Solutions Architect within the Client Services team, you act as trusted advisor, responsible for the technical requirements and end to end solutions integration of Wysdom cognitive services within the client’s environment.  You will work with one of the most diverse teams of linguists, data scientists, and innovators to deliver the best AI enabled customer experience.

Cognitive Data Specialist

As a Cognitive Data Specialist, you will be responsible for the performance of the AI and quality of the corpus and will focus primarily on the VA training.  You will work with the client as required to ensure corpus is performing in an optimal manner.

Conversational Experience Designer

As a Conversational Experience Designer, you will be responsible for the designs of the overall customer experience, including the end-to-end dialog flows & journeys of the solution ensuring  design leverages  UxD best practices for optimal customer experience.

Conversational AI Specialist

As a Conversational AI Optimization Specialist, your responsibility will be to help drive the success of our solution for our clients. This involves building conversation flows, performing AI training, and partnering with clients to enhance their deployments.

Conversational AI Lead

As a Conversational AI Lead, you will be responsible for leading all Conversational AI program activities.  You will work with all team members to ensure deliverables are completed on time, with high quality and exceeds client expectations and goals.

Program Director

Responsible for the overall success for the client, including the end-to-end delivery and optimization of the solution, you will manage the sales process from pillar to post, including technical and commercial proposals, pipeline management, sales forecasting, and contractual documentation.

David Trotter, Wysdom

David Trotter

SVP, Sales & Marketing

David has 30 years of global sales leadership experience as a collaborative leader who believes in a strong team concept within sales and marketing organizations. David has spent many years working with growth companies and enjoys being face to face with customers and partners to create solutions that have a lasting effect on the customer’s business environment. 


Prior to joining Wysdom, David was the Vice President, Sales at Scalepad, and previously spent 11 years as Vice President for Latin America and Asia Pacific for Absolute Software. He also held senior sales management positions at GE Capital and Clevest.  

Michel Benitah

VP, Optimization & Delivery

Michel has 20 years of experience in leading the successful delivery of Conversational AI and Natural Language Customer Care solutions to some of the largest financial, telco, healthcare, utilities, and retail enterprises throughout North America. 

 

Prior to joining Wysdom, Michel spent 20 years at Nuance Communications, holding senior management and leadership positions within the enterprise division, most recently as director of the Toronto office and professional services team.

Frederic Lam

SVP, Sales

Fred brings in 25 years of international experience in sales and business development across North America, the Caribbean, Asia-Pacific, Europe, and the Middle-East.

 

Prior to Wysdom.AI, he held sales leadership positions at Oracle, Redknee, and Movius/Glenayre, successfully growing revenues in both large and small organizations. Fred has also been involved in the start-up community in the earlier stages of his career as an Investment Manager with SP Capital and was an alternate director on a few investee companies.

Karen Chan

Chief Engineering Officer, Co-Founder

With 20 years of experience in software and mobile, Karen has held senior technical roles at 5 startups, including Wysdom.AI, Clickfree, Mobile Diagnostix (HP), Teamatic, and Virtualthere.

Karthik Balakrishnan

Chief Technology Officer

Karthik has over a decade of hands-on, proven global expertise in emerging technologies and implementing intricate platforms and solutions for telecommunications and enterprise during his time at Amdocs, with senior positions in their India, Cyprus, America, and Canada offices.

Nitin Singhal

Chief Operating Officer

Nitin has over 20 years of success in global executions of business technology, driving operational efficiency and digital scalability for some of the world’s largest enterprise clients. 

 

Nitin spent 16 years at Redknee holding executive positions in Research and Development, Customer Operations, Partner Alliances, and most recently as COO.

Jeff Brunet​

President, Co-Founder

Jeff has more than 20 years of experience in the startup world, founding and growing 4 software companies: AracNet, Mobile Diagnostix (HP), ClickFree, and Wysdom.AI. 

 

His in-depth understanding of software development and the challenges in making new technologies successful in the startup world prove invaluable as he serves on the boards of XMG, SurfEasy (Opera), Locationary (Apple), Groupie, and as an advisor to Pushlife (Google), LogMeIn (IPO) and HP. 

 

Jeff holds 23 issued patents in the wireless and consumer electronics spaces and is the lead inventor on 30+ pending patents.

Ian Collins​

CEO, Co-Founder

Ian has founded and grown 6 technology companies over the past 20 years, primarily in the enterprise software space including Wyrex, Mobile Diagnostix (HP), Clickfree, and most recently Wysdom.AI. 

 

Ian invests, mentors, and sits on the boards of several startups in the Toronto area.