Keyword-based or Natural Language Processing Chatbots: What Does it All Mean?

Most chatbot providers would like for you to believe that their chatbot is so advanced that a human wouldn't be able to tell the difference between it and an agent. If you're skeptical of this, you have every right to be. There are a lot of chatbots out there, but very few of them are advanced enough to replace any agents. How do you know what to look for when shopping for chatbots?

In the world of chatbots, there are basically two types: keyword-based chatbots and chatbots that leverage natural language processing (NLP).

Keyword-Based Chatbots

Keyword-based chatbots function like old-fashioned search engines: the customer types in a word or a phrase, and the bot matches that word or phrase with a pre-loaded response. For example, if a customer typed in, "Book a hotel room," the bot might return a link to a help center article about how to book a hotel room, but only if that input had been preloaded. The advantage to this is that it gives you tremendous control over your brand's automated messaging. The bot will never reply with content that has not been manually loaded into the system.

Keyword-based chatbots are limited by how manual their setup is. Unless a response has been programmed for them, the bot will not recognize misspelled words or slang. There are some with a little bit of machine learning that helps, but few that we've seen perform as well as you would hope. Moreover, they are only applicable within their own context. Ask the question "book a hotel room" to a chatbot for a bookstore and it will likely return books about hotels. The majority of the market is made up of keyword-based bots.

Natural Language Processing Chatbots

Alternatively, there are chatbot providers leveraging natural language processing (NLP), which allows the bot to utilize a contextual understanding of a question towards its resolution. Whereas a keyword-based bot can only identify preloaded words or phrases, NLP is programmed to consider the form of the sentence, its parts of speech, and any available context so that it can then apply that information to the response. For example, when given the inquiry "Book a Hotel Room," an NLP chabot would recognize that the user is trying to take an action that involves a location, and could then surface results based on available contextual information like similar actions taken in the past, a device's current location, or the price range of hotel rooms in a customer's account history.

Additionally, NLP chatbots have an understanding of the whole conversation and can ask follow-up questions to acquire additional context to even better tailor results. If all goes to plan, the end result should be a resolution tailored specifically to a customer, rendered in a fraction of the time it would take an agent.

NLP is ideal for companies that already leverage chat with experienced agents. Since NLP is a form of machine learning, most tools come equipped with functionality to instantaneously digest the entire chat history of an organization as training. This means within minutes, the bot can provide customers with the same responses they were historically getting from human agents, but in a fraction of the time.

Many providers advertise that their chatbots can learn to answer customer inquiries with a chat history of about ten thousand conversations, but the more data the machine learning engine has, the more applicable its responses will be. Be aware, however, that a bot is only as good as its trainers: historically bad agent responses will precipitate bad bot responses.

The questions we always get asked are, "How good are these chatbots?" and "Can they actually replace human agents?" Unfortunately, the answer isn't as simple as you might think. Keyword-based chatbots can be used to deflect some of your most basic questions like pricing, password resets, or other common issues, but for the best customer experience, you should always provide options for the customer to contact support if they get frustrated or need further assistance.

Similarly, most NLP chatbots aren't quite ready for primetime either. In most situations, agents and NLP-based chatbots work together. The bot reads the incoming message, provides a suggested response to the agent, and the agent either accepts or edits it before sending. Most NLP chatbots are built with a confidence rating based on the frequency that that response has been used on similar questions in the past. For example, let's say the bot is 90% confident that it has the right answer, it can automatically send a response, while anything with a lower confidence rating is routed to an agent. That's about as advanced as they get today.

So, which do you choose? It depends on the situation. Either way, chatbots are a great tool to implement, but impossible to be successful without human involvement.