The problem
A retailer or software company writes help articles and still gets tickets asking questions that have been answered before. Help centres in Zendesk or Intercom suggest articles as the customer types, and companies report very different deflection rates, from single figures to most enquiries, depending on how well the suggestion matches the question.
I measured how often a suggestion step that searches every answered question by words and meaning puts a useful answer in front of the customer before the ticket is sent.
- Team
- Support operations lead, knowledge manager, agent teams across shifts
- Systems
- Zendesk Guide, Salesforce Service Cloud, Freshdesk or Intercom; the order or billing system for account context
- Measure
- Share of new questions shown a right answer before they become tickets; tickets per hundred orders
What I built
A suggestion step on the contact form and in the agent view. As the customer types a subject, it searches every answered question and article, both by the words used and by meaning, and shows the closest answered ones. When the match is weak it shows nothing and the ticket goes through as usual.
Results
For 375 new questions that community moderators had marked as duplicates of an earlier one, searched against the full library of 167,765 questions, the hybrid search puts the answered duplicate in the first five suggestions 67.7% of the time (95% range 63.2% to 72.3%). Keyword search on titles, which most help-centre search boxes use, manages 48.0%.
| Method | First suggestion right | Right in top five | 95% range, top five | Recall in top ten | MRR@10 |
|---|---|---|---|---|---|
| Keyword search on titles | 25.1% | 48.0% | 42.9% to 53.1% | 20.8% | 0.346 |
| Keyword search on full text | 31.7% | 56.8% | 52.0% to 61.9% | 29.4% | 0.423 |
| Meaning-based search | 30.4% | 54.7% | 49.6% to 59.5% | 29.7% | 0.410 |
| Hybrid of both | 40.0% | 67.7% | 63.2% to 72.3% | 40.2% | 0.523 |
Showing suggestions only when they are likely right
What meaning-based search adds
Customers describe the same problem in different words. In these pairs the new question and its answered duplicate share at most one word, yet the hybrid search put the duplicate first.
| New question | Answered question it was matched to |
|---|---|
| Why doesn't ubuntu 12.04 upgrade libreoffice, as firefox? | How do i install the latest stable version of libreoffice? |
| Automatic updates | Update manager problem [requires installation of untrusted packages] |
| How to change the color tone of the ambiance theme? | Changing the ubuntu 13.10 ambiance colours |
| Howto automatically remove older package versions from a custom repository? | Delete old packages from local repository |
Data and method
- Data
- AskUbuntu duplicate-question benchmark (Lei et al., NAACL 2016; dos Santos et al., 2015; CC BY-SA): 167,765 questions from the AskUbuntu forum. The evaluation questions are the published development and test questions whose duplicates were marked by the community
- Search space
- The full question library, not a short candidate list, which is the harder and more realistic setting
- Methods
- BM25 keyword search on titles and on title plus body; meaning-based search with BAAI bge-small-en-v1.5 embeddings of titles; a hybrid using reciprocal rank fusion of full-text keyword search and embeddings
- Measures
- Duplicate in first place, in the top five and recall in the top ten; bootstrap 95% range on the top-five figure
- Suggestion bar
- Suggestions shown only when the top embedding similarity clears a percentile of all queries
Limitations
- The questions are technical Linux questions from a public forum. A store’s questions about orders, returns and sizing are shorter and repeat more, which usually makes matching easier, but this is measured on the business’s own tickets first.
- A marked duplicate shows that an answer exists, not that the customer would accept it. The live measure is the share of customers who leave the form without submitting and do not come back within a week.
- Questions that need order or account details, such as where is my parcel, are better answered by an order look-up than by an article.
- Help-centre content must be current; out-of-date articles are flagged when agents keep overriding a suggestion.
How I would take it live
- Export a year of tickets and the help-centre articles from Zendesk, Freshdesk or Gorgias
- Measure how many tickets match an existing answer, and write the missing articles first
- Add suggestions to the agent view for two weeks, then to the contact form above the agreed bar
- Report weekly on form exits without a ticket and on repeat contacts