Want to see how we do chatbot analytics? Join us for a product demo webinar.

Blog
An image of chatbot assembly that also include bot analytics to measure performance quality

The world of chatbots and Conversational AI is evolving rapidly, with numerous companies offering an array of solutions or incorporating them into their products and services. As a business owner, you may find yourself underwhelmed with the analytics available in these platforms.

Bot platforms typically provide limited access to the data, surface-level reporting with metrics for volume, containment, intent metrics. Chatbot teams supplement their efforts with fragmented tools and manual reporting using tools like SQL queries, manual search, Tableau…

Chatbot-native reporting leave you feeling unfulfilled, as you struggle to answer basic questions: 

  • What are my customers talking to the bot about? 
  • How effective is the bot at answering their queries? 
  • Do my customers like the bot experience? 
  • Is the bot helping the business save (or make) money; how can I prove it?
  • What improvements do I prioritize?
  • What metrics should I be sharing with my key stakeholders?

And suddenly, you find yourself thinking about building chatbot analytics software. 

But, hold on; before you pull the trigger on the time and resources needed to build an in-house bot analytics solution, consider these six points.

1. Key features for chatbot analytics software

You need to define they key list of features that should be built into your analytics softwareWhile it’s important to think in the short term, and identify features to get off the ground as a minimum viable product (MVP), it’s more important for the team to have a clear vision of how the product is going to look after a few iterations. A team without an extremely strong background in chatbots and Conversational AI will always struggle to understand the features that might be needed in the future. Building a comprehensive vision for a product is the key to the success of the product. A shortlist of features must include: 

Conversation topics: effective bot analytics must be able to by extracting “primary contact reasons” from customer interactions with both your chatbot and live agents. By utilizing AI techniques like topic modeling, you should be able to identify the real reasons your customers are engaging with you, over and above Intents, which is what the business has already identified. However, achieving this level of insight requires AI expertise including model building and training. 

Automation: Accurately measuring how well the bot is automating the conversation goes beyond simple containment metrics. Bot platforms often miss this metric, taking into account only whether the conversation was contained, however this measure overlooks whether or not the conversation met the customer’s needs and satisfaction. To accurately assess automation AND customer satisfaction, you need specific “signals”, which requires additional AI model development and training. Learn more about containment in the article Demystifying bot containment with chatbot analytics.

Customer Satisfaction: In order to measure customer satisfaction many platforms turn to customer surveys, but often the participation rate is less than 5% and significantly skewed towards negative feedback. Clearly, this approach is far from ideal. Instead, creating an effective customer satisfaction measurement encompassing 100% of conversations demands a combination of innovative signals and advanced AI models to generate the necessary inputs. You can read more about this innovative approach in this article about how to measure CSAT without surveys. Achieving this level of intelligence requires strong AI and NLU expertise that would need to be hired or developed internally.

New intent discovery: Conversational bots are built using natural language understanding and intents as the core mechanism to match customer needs with bot responses. However, there will always be certain things that the bot doesn’t understand, and it’s crucial to keep track of these.  

Using an external topic modeler, you can identify topics in conversation data that are missing from the existing intents. Even better, it can be configured to run on a 24-hour basis, uncovering new intents, or topics, as they emerge. This feature is highly valuable as it will enable you to expand their intent repository, and capture more of the conversations their customers want to have, thus improving the performance of their bot.

False positive identification: NLU-based bots may occasionally make mistakes, incorrectly understanding customer needs and following the wrong path. Bot platforms typically don’t provide false positive identification. To address this issue, you need a custom-built AI technique separate from NLU/classifier design.

Financial impact: Understanding the financial impact of your chatbot program is essential to justify past and future investments. Early-stage financial metrics may not be clear, and bot platforms might not readily disclose this information. Being able to connect the dots between contact center savings, cost per bot conversation, and the impact of bot improvements on both of these measures will inform sound business decisions around the cost of Conversational AI.

2. Integrating with data systems

Building a robust bot analytics engine requires connectivity to multiple data sources: think chatbot data from multiple platforms or services, voice data, live agent data, etc. Understanding and deciding how the data will be stored, accessed, and reported is not an easy task.

Accessing and normalizing conversation data: Each chatbot platform utilizes different log structures and reports on varying events, requiring you to access and normalize the data to ensure consistency. As bot platforms typically don’t look beyond the bot, you’ll need to build a system that follows conversations from start to finish. Since bot platforms primarily concentrate on the chatbot’s performance, they often neglect the whole customer journey, thereby missing crucial insights. To gain a comprehensive understanding of customer interactions and satisfaction, it is necessary to build a system that tracks the entire conversation, including any escalations to live agents or subsequent follow-ups.

Considerations for data integration: When deciding on an approach to data integration, keep the following questions in mind:

  • Where is your conversation data currently stored, and where will it be in the future?
  • Do you plan to expand to multiple chatbots, including voicebots?
  • Will you store conversation data in a data cloud like Snowflake, BigQuery, or Amazon Redshift, or leave it in log files within the bot and customer care platforms?

Maintaining connectors: Once you’ve established integrations with conversation data sources, it’s crucial to consider the ongoing maintenance of connectors, as source platforms will continuously change.

Choosing between internal teams and vendors: When deciding whether to rely on an internal team or a vendor with pre-built connectors, consider the following:

  • Scalability: Vendors with existing partnerships and technology expertise can better integrate and maintain conversation streams with analytics and reporting suites.
  • Adaptability: Off-the-shelf analytics tools are designed to handle various interaction scenarios, enabling smoother integrations as your chatbot needs evolve.

Early stage decisions will determine future scalability, including connectivity, storage, deployment and scaling. Poor initial choices can lead to system bottlenecks, data fragmentation, and other challenges that hamper your ability to scale effectively. 

3. Performance

Running analytics on thousands of multilingual conversations can be a resource-intensive task, demanding powerful AI models and extensive cloud services. To achieve effective chatbot analysis, it is vital to strike the right balance between cost and insight. Managing analytics for multilingual chatbot interactions requires resources including sophisticated AI models and costly cloud services. Balancing model usage is a critical aspect of avoiding exorbitant expenses while simultaneously capturing essential insights from conversation data.

Equally important, you need to evaluate factors like the depth of analysis, the frequency at which the model is run, and the need for real-time analysis. While basic analytics may seem adequate in the early days, over time, the need will expand and the questions about bot performance will become more sophisticated, resulting in more intensive analyses. Will an internal team be as well equipped as an experienced vendor to deliver the desired performance?

4. Chatbot analytics time-to-value

It is well known that IT projects often exceed their estimated timelines and costs. In fact, an older report by Gallup that still rings true today shares that one in six IT projects exceeds their estimated timeline, with an average cost overrun of up to 200%. In the world of chatbots, it is essential for businesses to grasp the consequences of postponing analytics. 

Failing to get a handle on bot performance, quickly, will adversely affect businesses that depend on these bots for a multitude of customer interactions. 

If your chatbot has been in-market for 3-6 months already, you probably need performance insights yesterday. Building your analytics solutions from scratch will certainly lead to delay, and will lead to lost revenue, savings, and customer loyalty. 

5. User training and support

The effectiveness of analytics and reporting solutions relies on a team’s ability to learn the tools and extract desired results. These solutions can be complex, requiring comprehensive training programs to maximize insights and chatbot improvement. Good documentation, training, and ongoing product support are crucial for continued success, especially in the face of team turnover. In the long run, external vendors dedicated to analytics platforms may offer more extensive and up-to-date documentation and support for your solution.

6. Cost of building your own chatbot analytics platform

In determining whether to build or buy analytics and reporting software, it’s crucial to consider development, maintenance, and upgrade costs. Creating a simple spreadsheet to compare costs can aid in understanding which option is more cost-effective. Generally, external vendors building solutions for multiple customers offer more affordable options due to cost amortization, as opposed to internal teams that work at cost for a single customer’s project.

As businesses consider implementing chatbots into their products or services, determining the return on investment (ROI) becomes essential. 

Conclusion

When it comes to managing a chatbot program, incorporating analytics is undeniably crucial. The idea of building your own bot analytics system might seem appealing, but it’s important to consider the significant investment of time and resources it entails. 

For those with deep domain expertise in Natural Language Processing (NLP), experience working on multi-lingual large language models like GPT and BERT, and AI model training and productionizing in general, building a chatbot analytics platform in-house might be the way to go. However, for the majority, third-party options like the Wysdom Operations Center will be a ready-to-use solution, at a more attractive price point, delivering state-of-the-art analytics.

Subscribe to our newsletter

Get inspired

Read more insights on the latest in virtual agent analytics and performance management

Subscribe to our newsletter

We use cookies to ensure that we give you the best experience on our website. By clicking “I Accept” or if you continue to this site, we will assume that you consent to the use of cookies unless you have disabled them.