Manaz Afzal
Retail & SaaS · Help-centre deflection

The answer the customer was about to ask for, shown before the ticket

A suggestion step that searches every answered question by words and meaning, tested on 375 new questions against a library of 167,765.

68%answered duplicate in the top five; title search 48%
40%right answer as the first suggestion
74%right in the top five when shown for the strongest half

Executive summary

Searching answered questions by words and meaning shows the right existing answer in the first five suggestions for 68% of new questions, against 48% for the title keyword search most help centres use.

68%of new questions shown their answered duplicate in the top five, against 48%
40%with the right answer as the first suggestion
221 hof agent time saved a month at 20,000 tickets

Situation

Many new tickets ask something already answered; customers miss the answer because they use different words from the article.

Approach

BM25 keyword search, embeddings and a hybrid of both, tested on 375 new questions with marked duplicates against the full library of 167,765 answered questions.

Result

The hybrid finds the answered duplicate in the top five 67.7% of the time, against 48.0% for title keywords. Suggestions are held back when the match is weak.

The problem

A retailer or software company writes help articles and still gets tickets asking questions that have been answered before. Help centres in Zendesk or Intercom suggest articles as the customer types, and companies report very different deflection rates, from single figures to most enquiries, depending on how well the suggestion matches the question.

I measured how often a suggestion step that searches every answered question by words and meaning puts a useful answer in front of the customer before the ticket is sent.

Team
Support operations lead, knowledge manager, agent teams across shifts
Systems
Zendesk Guide, Salesforce Service Cloud, Freshdesk or Intercom; the order or billing system for account context
Measure
Share of new questions shown a right answer before they become tickets; tickets per hundred orders

What I built

A suggestion step on the contact form and in the agent view. As the customer types a subject, it searches every answered question and article, both by the words used and by meaning, and shows the closest answered ones. When the match is weak it shows nothing and the ticket goes through as usual.

QuestionCustomer typesa subjectSearchWords and meaningacross the libraryStrong matchAnswered questionsshown firstWeak matchTicket created,agent sees matches
Agents see the same matches beside each ticket that does come in, so a reply can link the existing answer.

Results

For 375 new questions that community moderators had marked as duplicates of an earlier one, searched against the full library of 167,765 questions, the hybrid search puts the answered duplicate in the first five suggestions 67.7% of the time (95% range 63.2% to 72.3%). Keyword search on titles, which most help-centre search boxes use, manages 48.0%.

Keyword search on titles48%Keyword search on full text57%Meaning-based search55%Hybrid of both68%
Keyword search on titles25%Keyword search on full text32%Meaning-based search30%Hybrid of both40%
The first suggestion alone is the right one 40% of the time for the hybrid search.
MethodFirst suggestion rightRight in top five95% range, top fiveRecall in top tenMRR@10
Keyword search on titles25.1%48.0%42.9% to 53.1%20.8%0.346
Keyword search on full text31.7%56.8%52.0% to 61.9%29.4%0.423
Meaning-based search30.4%54.7%49.6% to 59.5%29.7%0.410
Hybrid of both40.0%67.7%63.2% to 72.3%40.2%0.523

Showing suggestions only when they are likely right

50%60%70%80%90%100%10%20%30%40%50%60%70%80%90%100%Right answer in top five, of those shownShare of new questions shown suggestions
Showing suggestions for the half of questions with the strongest match, the right answer is in the top five 74% of the time. For the rest the form stays out of the way.

What meaning-based search adds

Customers describe the same problem in different words. In these pairs the new question and its answered duplicate share at most one word, yet the hybrid search put the duplicate first.

New questionAnswered question it was matched to
Why doesn't ubuntu 12.04 upgrade libreoffice, as firefox?How do i install the latest stable version of libreoffice?
Automatic updatesUpdate manager problem [requires installation of untrusted packages]
How to change the color tone of the ambiance theme?Changing the ubuntu 13.10 ambiance colours
Howto automatically remove older package versions from a custom repository?Delete old packages from local repository

Data and method

Data
AskUbuntu duplicate-question benchmark (Lei et al., NAACL 2016; dos Santos et al., 2015; CC BY-SA): 167,765 questions from the AskUbuntu forum. The evaluation questions are the published development and test questions whose duplicates were marked by the community
Search space
The full question library, not a short candidate list, which is the harder and more realistic setting
Methods
BM25 keyword search on titles and on title plus body; meaning-based search with BAAI bge-small-en-v1.5 embeddings of titles; a hybrid using reciprocal rank fusion of full-text keyword search and embeddings
Measures
Duplicate in first place, in the top five and recall in the top ten; bootstrap 95% range on the top-five figure
Suggestion bar
Suggestions shown only when the top embedding similarity clears a percentile of all queries

Limitations

  • The questions are technical Linux questions from a public forum. A store’s questions about orders, returns and sizing are shorter and repeat more, which usually makes matching easier, but this is measured on the business’s own tickets first.
  • A marked duplicate shows that an answer exists, not that the customer would accept it. The live measure is the share of customers who leave the form without submitting and do not come back within a week.
  • Questions that need order or account details, such as where is my parcel, are better answered by an order look-up than by an article.
  • Help-centre content must be current; out-of-date articles are flagged when agents keep overriding a suggestion.

How I would take it live

  • Export a year of tickets and the help-centre articles from Zendesk, Freshdesk or Gorgias
  • Measure how many tickets match an existing answer, and write the missing articles first
  • Add suggestions to the agent view for two weeks, then to the contact form above the agreed bar
  • Report weekly on form exits without a ticket and on repeat contacts

Financial impact

About 1,897 tickets a month answered before they are submitted

The saving is agent time on tickets that already had an answer. How many customers accept a suggestion is an assumption until the form is live.

InputValueSource
New tickets a month20,000 national retailer or SaaS business
Share already answered somewhere35% measured from the ticket export in week one
Right answer in the top five67.7%Measured hybrid search
Customers who accept a right suggestion40%
Agent minutes per ticket7
Loaded cost per hourA$40 support agent

Calculation

  1. Tickets answered before submission, title search1,344
  2. Tickets answered before submission, hybrid1,897
  3. Agent time saved a month221 hours
  4. Annual valueA$106,206

Annual value of agent time savedA$106,206

Faster answers for customers and fewer repeat contacts are further gains and are not counted here.

Implementation and rollout

Five weeks from ticket export to suggestions on the contact form

  1. Week 1

    Export and measure

    A year of tickets and every help-centre article from Zendesk, Freshdesk or Gorgias; measure the share that match an existing answer.

    Support lead, Manaz

  2. Week 2

    Fill the gaps

    Write or update the articles behind the most frequent unanswered questions.

    Support lead

  3. Weeks 3 to 4

    Agent view

    Matches shown beside each incoming ticket; agents link the existing answer and flag bad matches.

    Agents

  4. Week 5 onward

    Contact form

    Suggestions shown on the form above the agreed bar. Weekly report on form exits without a ticket and repeat contacts.

    Support lead, Manaz on support

Tools

Zendesk Guide, Salesforce Service Cloud, Freshdesk or Intercom, Shopify or Stripe for order and account look-ups, Embedding search service with BM25, hosted in the business’s own cloud account, A small widget on the contact form

Team

  • Support lead owns the help centre and the suggestion bar
  • Agents flag wrong matches and missing articles
  • Manaz search build, integration and monitoring

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
A wrong answer is suggestedSuggestions are shown only above a bar, and the customer can always submit the ticket.
Out-of-date articlesArticles agents override often are flagged for review each week.
Order-specific questionsWhere-is-my-order questions go to an order look-up, not an article.
Customer data in ticketsTicket text is indexed inside the business’s own account; no data goes to outside services.

Key takeaways

Key takeaways

  • Title keyword search finds the answered duplicate in the top five 48% of the time; the hybrid lifts that to 68%.
  • Keywords and meaning each find duplicates the other misses, which is why the hybrid wins.
  • Suggestions are held back when the match is weak, so the form stays out of the way.
  • The first step is measuring how many tickets already have an answer, from the business’s own export.