Wynston Loading Icon

Aug 20, 2019 | By Brent | Share this article

The majority of B2C enterprises are now working on deploying Conversational AI products, for both internal and external facing use cases.  With the relative immaturity of the industry, companies are still being caught off guard with the amount of work that can be required to deliver a successful virtual assistant.

To speed up sales, many vendors are selling on a message of “this isn’t any work at all” which is obviously not true. This is the snake oil of Conversational AI and is hurting the whole industry.  By going into a project without the proper expectations, companies are being caught off guard with the amount of effort required to actually deliver a great customer experience.

Here are the top 3 pitfalls to avoid when starting a Conversational AI project.

1. Forgetting to gather system feedback or “close the loop”

One of the ways AI in the enterprise is different from all of the enterprise software delivered in the last 20-30 years is that it requires a feedback mechanism.  Whatever function you are using, AI will start off at some rate of success and will improve over time. This is very different from the software world we’ve all become accustomed to.  In the traditional software world, we gathered requirements, built the best product in a certain amount of time, tested and released it into a production environment. Now, in the Conversational AI world, it is very different.  

In AI, you create a deep learning model that will make a prediction to support your business.  For example, a model that determines if you should make a car loan to an applicant would predict the probability of them paying you back..  In the first iterations of the model created to make this prediction, the performance will be weak. You will then optimize the model, feeding more training data and adding pre and post processes to improve accuracy.  The key to AI is that this process will never end. You will reach a point where the predictions are accurate enough to release your AI to a production world, but that isn’t the end of optimization.

To enable this AI “training” in the enterprise, you need a closed loop system.  You need to find clues that may not be explicit in order to tell you if the model succeeded.  If you make tens of thousands of predictions with your model, and have the ability to know the accuracy of these predictions, you can use that feedback to drive continuous improvement.

If not considered prior to your system deployment, this will add considerable cost that could have easily been avoided.  This isn’t needed in traditional software deployments, so is overlooked many times.

2. Not analyzing the data from the AI feedback loop

At this stage of a deployment, where you have a mechanism to automatically take “clues” from your system and use that to determine if the model succeeded, you can also burn a lot of cycles.  In many cases, this feedback loop data isn’t clear and you may need to build additional models to cluster and analyze it. You’ll need to wrap the core model with other operational data to determine how to improve.

Let’s go back to the above example in reference to making loans to buy cars – in this particular scenario, you may be presented with data showing many default fields, and then add all of the other customer profile information that is available.  There will be thousands of opportunities to improve the core prediction model, but where should you start? What you need is a clustering system that will identify prevalent failure patterns that you should focus on eliminating from your prediction model.  We call these “trainable events”.

Developing a system that automates as much as possible without creating operation overhead will take time and another type of resource.  If you don’t have this planned from the beginning, you will be caught off guard and spend more time and money than necessary.

3. Failing to tune & test the models

You’ve finally reached the part of AI where you actually retrain your models.  You’ve closed the loop and built practical analytics to lead you to the most effective changes you can make.  Now, you want to influence as many model changes as possible without moving the performance backwards. This is one of the more difficult parts of using deep learning. In some cases, when you make changes to a model or introduce new training data, the system will actually perform worse than before!

To solve this problem, you’ll need tools that let you efficiently make changes and then run automated testing cycles.  This will allow you to make many changes and determine what impact they will have on performance. This must be done quickly and frequently. Using the right tools will enable an efficient training and testing cycle.

Anyone who has worked in enterprise software knows that delivering a quality system can consume a lot of QA resources.  To maintain the speed of system improvement and avoid wasting a pile of money , you’ll need to have planned and built these systems long before they are needed. This is the third pitfall to avoid if you want to keep your Conversational AI system learning and satisfying customers.

This all leads to what we call “AI Operations”. At Wysdom, our tools and experienced AI supervision team can efficiently manage an AI application.  This is rarely mastered by a new team within an organization that has expertise in other areas. 

To find out more about common AI pitfalls and how to prevent them, download our Whitepaper: 5 Reasons Conversational AI Fails and One Way to Guarantee Success