Chatbots: From understanding text to understanding customers

Chatbots: From understanding text to understanding customers

Dr. Andy Lewis,

Over the past few weeks, a Twitter joke was doing the rounds about Natural Language Processing (NLP):


Natural Language Processing!


Sorry, when do we want what?

I am sure we all recognise the frustration we often find when engaging with AIs as they try to understand us. In the past few years, speech response systems have become ubiquitous – they are in our phones, kitchens, and cars, helping us to navigate, find restaurants, and play our favorite music.

And, yes, we do marvel at them, but also get extremely frustrated as they fail to do the sort of things humans find simple. When I ask, “Do I need an umbrella in Boston right now?” my Amazon Echo remarkably tells me I do not need one, but fails to note that I am not in Massachusetts where the weather is fine, but in Lincolnshire, England where the rain is streaming down. Very few of my human associates would make such a mistake.

Your customers may have already interacted with a chatbot you have deployed to attempt to get answers to queries or even with help buying something. But you can bet they know they are engaging with a bot and not a human. Chatbots can appear clunky, limited, and be quick to give up. Most importantly, they show no understanding of the customer as an individual. They treat all customers the same, responding in the same way to all, lacking any contextual awareness of what the customer is trying to do. Can you be sure that your chatbot is improving your customer’s experience? Or is it just one more step in the way of the customer getting what they really want or need?

If your customer can easily spot they are talking to a chatbot because it cannot contextualise what is said to them, then your chatbot lacks customer empathy and should be retired.

NLP lacks contextual understanding

Typically, in Natural Language Processing (NLP), words are treated as the sole data input in each individual processing request. NLP tries to extract the topics, intents, entities (names, account numbers, etc.), and sentiments from this text. But often, there is not enough information to fully express the complete meaning of the sentence. This is because meaning is wrapped up in the context of the conversation. By just looking at words alone, the meaning is often lost, ambiguities persist, and errors are made. To be truly intelligent, a chatbot needs to do more than just process words, it needs to maintain stateful information and inject external contextual understanding and insight. In my Boston weather example above, the essential external contextual data concerning my current location was absent, resulting in entirely the wrong response.

Which brings us to a huge problem when we want to deploy NLP technologies to improve customer experience. NLP has customer experience applications in chatbots, email handling, social media monitoring, IVRs and call center assistance. In each of these areas, we want to automate as much customer interaction as possible, ask the relevant questions, and initiate the right actions, while maintaining great customer experiences. Poorly understood requests and inappropriate and unpersonalized responses degrade that experience.

NLP technologies only process the words presented, but organizations want and need more. They want to understand the customer’s holistic context, full intent, and best action to take. They want to engage in a way that is responsive, consistent, timely, and accurate. To do that, you need to personalize the response depending on the customer’s segment, life-time value, business issue, and stage in their journey. None of this vital context is in the text. So how do we blend customer insight and data with the presented text to extract deep meaning and insight, and transform this engagement into valuable action? The solution is combining context with intelligence.

The customer is the context

Context demands we bring into the mix customer memory, insight, predictions, rules, and a learning capability. Real-time Next Best Action (NBA) technologies are designed to process customer data and context to recommend the next action that will produce the best business value within that interaction. Combining NBA with NLP can produce an intelligence that contextualizes the words said, and predicts the most relevant and timely responses – like the next best question to ask; or the next best case information to offer.

Context allows the chatbot to hand off the customer to the most appropriate agent to manage a request. If a customer says, “I want to close my account,” then the action the chatbot takes will depend on if the customer has recently made a complaint and has a high churn risk, or is merely closing an old, unused facility.

To evolve, chatbots need to learn from contextual data

In order to gain deep empathy with customers, the chatbot needs to learn from each customer interaction. Given any NLP classification of a request, an NBA engine may have many possible responses to that request. Predictive models can be used to rank competing responses according to their likelihood of a positive business outcome. Using automated machine learning within the chatbot allows responses to improve over time as the chatbot becomes more experienced.

When customer context is taken into account, predictive models can apply outputs of textual analysis, such as conversation categories, intents and sentiments – plus combine these with customer data, product and transactional data, and real-time contextual data, like weather, location, and recent activity. This is the path to true personalization and accurate, relevant responses.

Does Your Chatbot Dream of Electric Customers?

So if we have many competing responses to a request, our empathic chatbot needs to understand if a chosen response was successful or not, and then spend some time dreaming of our customers – that is, analyzing responses, processing to understand why responses worked or did not, and then evolving to a better and more human chatbot.

However, given that we may be managing many hundreds of potential responses to customer requests, relying on manual human intervention to help chatbots learn will be cost-prohibitive. The chatbot needs to rely on automated machine learning, managing hundreds, if not thousands of predictions, and should be continuously building new predictive models in response to each customer interaction.

Our chatbot can learn how words are used in the context of your business and the specific intent for each individual customer within the context of their interaction. But to truly create a seamless experience for a customer, this intelligence needs to extend across channels. At some point the chatbot will want to conclude the conversation and, maybe, hand off to another channel or system. Unless that channel is also driven by the same Next Best Action contextual awareness, the customer will experience a disjoin and your good work may be undone.

The reverse is true, too: The chatbot should be learning from interactions in other channels. A common “brain” is essential for this. It provides insight across all of your customers’ experiences. For example, if a telco chatbot is asked what phone upgrade offer is best, the chatbot needs to draw on real-time learnings from other channels about what handsets and tariffs are best suited for this customer.

Human-like empathy is essential

Getting this right means chatbots will be able to offer customer experiences that are nearly indistinguishable from those presented by your best human agents now. Processing words is not enough. Electric empathy is the key. To be successful we need to fully focus on the customer, their context, their personal history, and not just the words they are saying.

Learn More: Find out how an intelligent virtual assistant can deliver great customer service with dynamic conversations on any channel.


  • Tema: Automatización de marketing
  • Tema: Excelencia operativa
  • Tema: Inteligencia artificial
  • Desafío: Automatización de procesos digitales
  • Desafío: Marketing personalizado
  • Área de producto: Marketing
  • Área de producto: Customer Decision Hub
  • Área de producto: Customer Service

About the Author

Dr. Andy Lewis, a principal solutions consultant for AI and decisioning, is changing the world for the better through real-time, contextual, AI-driven next-best-action conversations.