Beyond “Sorry, I Didn’t Understand” – How to Handle Chatbot Errors

Last week, I was ordering some supplements online. I had a question about one of the products, so I did what we are increasingly encouraged to do: I asked the chatbot. After a couple of fruitless interactions, the chatbot eventually told me that it could not answer questions about the products (only about orders); and told me to ask a sales assistant at one of their stores, which, let’s face it, was not particularly helpful.

I’m sure that I am not alone: No matter how capable chatbots become, there are still many situations where they do not understand what the customer wants, or where they are unable to deal with the request. So, if a firm can’t avoid chatbot failures, can we, at least, reduce the negative impact of such failures?

This is the problem that Ulrich Gnewuc and Fabian Reinkemeier explored in the paper “Overcoming Breakdowns in Customer-Chatbot Interaction: Design and Impact of Collaborative Repair Strategies”, published in MIS Quarterly.

The researchers were particularly interested in the potential of what they called collaborative repair strategies. The idea is quite intuitive: When two persons misunderstand each other, they work together to establish what went wrong and try again. Yet when there are problems in chatbot interactions, the strategy is for the chatbot to either push the customer to solve the situation (e.g., the chatbot might say: “Sorry, I didn’t understand that. Please try again.”) or to try to find a solution, through a series of trial and error.

So, Gnewuch and Reinkemeier designed a repair strategy which emulates the way humans interact (i.e., the chatbot and the customer work together to resolve the breakdown in communication).

The researchers analysed more than 21,000 real customer-chatbot interactions, and identified four main types of breakdown:

• The request is too complex
• The request is unclear
• The request is too short
• The request follows an unexpected format

Then, the researchers designed a repair strategy in which the chatbot helps the customer understand what work needs to be done, and which includes:

1. Diagnosing which type of breakdown had taken place (e.g., request is too complex)
2. Explaining the problem to the customer (e.g., I struggle with long requests)
3. Providing guidance to the customer on how to (re)formulate the request, depending on the type of problem (e.g., focus in key information)

For example:

The collaborative strategy tested by Gnewuch and Reinkemeier increased the proportion of breakdowns that were ultimately resolved from 32 out 100 when the non-collaborative approach was used, to 38 out of 100. That’s an improvement of around 18%.

Customers were also less likely to abandon the chatbot immediately after a breakdown, when using the collaborative strategy 41.7% vs 51.0% when not using it. And customer feedback about the repair messages was less negative in the collaborative than the non-collaborative condition.

This is a valuable finding for managers: It is important to develop a recovery strategy that doesn’t put the sole responsibility for solving the problem on the customer. A better approach is to explain what went wrong, and to provide useful guidance about what to do next.

However, there is another finding that managers ought to take notice of: customer satisfaction remained relatively low (namely, 2.07 out of 5 in the collaborative condition, compared with 1.82 in the non-collaborative one). That is, repairing the conversation through a collaborative strategy mitigates the damage caused by the chatbot limitations, but does not eliminate it. So, when evaluating a chatbot, consider not only whether it reduces the likelihood that the customer will abandon the interaction, but also whether it actually helps the customer achieve what they want to use the website for.