Manaz Afzal
Financial services · Inbox triage

Complaints sorted to the right team, with the unclear ones left for a person

A triage step for a shared complaints inbox, tested on 111,466 later complaints.

88.2%routed to the right team; keyword rules 56%
83%sent automatically, at 93.8% accuracy
19,080held for a person
0.74macro F1 across nine teams

Executive summary

A triage step on a shared complaints inbox routes 88% of complaints to the right team, against 56% for keyword rules, and leaves the unclear 17% for a person.

88.2%routed to the right team on 111,466 later complaints (keyword rules 56.2%)
83%sent automatically at 93.8% accuracy; 19,080 held for a person
0.74macro F1 across nine teams (95% CI 0.737 to 0.747)

Situation

Complaints arrive through a web form and a shared inbox. A triage officer reads each one only to forward it, keyword rules send almost half to the wrong place, and each wrong hop eats into the RG 271 response deadline.

Approach

Logistic regression on 135,341 redacted CFPB complaints from July 2022 to June 2023, with the routing bar chosen on July to September and tested once on October to December 2023.

Result

Routed mail is 93.8% right; held mail would have been 61% right, which is why a person reads it. A random split would have reported macro F1 0.827 against 0.742 on the time split.

The business problem

Under ASIC’s RG 271 a lender has to acknowledge a complaint within a business day and give a written response within 30 calendar days, and systemic issues have to be spotted early. In most complaints teams the first step is manual: someone reads the email or form, picks a product and issue category, and passes it to the team that owns it. When that first read is wrong, the complaint bounces between teams and the 30 days keep running.

I built a routing step that reads each complaint as the customer wrote it, sends it to the owning team, and holds anything it is unsure of for a complaints officer, then measured how often it gets the first hand-off right against the keyword rules many teams start with.

Team
One or two triage officers; five to ten specialist queues
Scale
Hundreds to thousands of complaints a month
Systems
Outlook or a shared complaints inbox, Salesforce Service Cloud, Dynamics 365 or a complaints system such as Complaints Pro
Constraint
Personal information in free text; Privacy Act; AFCA reporting

What was built

A triage step on the shared inbox. It reads each redacted complaint, scores all nine teams at once, sends the clear ones straight to a queue and leaves the rest for a person. Every correction is saved and the model is retrained monthly.

ArrivesWeb form or email,personal details redactedReadText model scoresnine teamsSure enoughScore 0.80 or more:to the team (83%)Not sureBelow 0.80: a personpicks (17%)LearnCorrections saved,monthly retrain
Triage workflow in the help desk. Nothing is answered automatically; only the queue is chosen.

Results

Credit reports78,600Cards8,970Debt collection7,933Bank accounts5,900Mortgages2,838Student loans2,794Money transfer1,982Vehicle loans1,509Personal loans940Held for a person19,080
Credit reports dominate the caseload. 19,080 complaints were held for a person at the 0.80 bar. Data: CFPB Consumer Complaint Database
Keyword rulesModelCredit reports96%Cards68%Debt collection59%Bank accounts85%Mortgages86%Student loans81%Money transfer61%Vehicle loans60%Personal loans46%
The model routes 88.2% to the right team against 56% for keyword rules. Personal loans and debt collection are weakest: those letters read like card or credit-report complaints.
MethodAccuracyMacro F1Balanced accuracy
Keyword rules56.2%0.41544.3%
Send all to the biggest team70.5%0.09211.1%
Text model88.2%0.742 (0.737–0.747)71.3%
40%60%80%100%0.30.40.50.60.70.80.91.0Accuracy on routed mailShare sent automatically0.80: 83% sent, 93.8% rightConfidence bar
The bar was chosen on July to September for 95% accuracy on routed mail and checked once on October to December: 93.8% on the 83% sent automatically. Held mail would have been 61% right, which is why a person reads it.
<50 words22,564 complaints84%50–99 words26,194 complaints90%100–199 words34,210 complaints90%200–399 words19,321 complaints88%400–799 words7,059 complaints88%800+ words2,118 complaints89%
Very short complaints are harder; length beyond 50 words makes little difference.

Data and method

Source
US Consumer Financial Protection Bureau, Consumer Complaint Database (public, narratives redacted by the CFPB)
Window
665,936 complaints with text, July 2022 to December 2023
Split
Train to June 2023 (135,341 used), choose the bar on July to September, test once on 111,466 complaints from October to December
Teams
CFPB products mapped to nine queues; product renames in 2023 merged

Logistic regression on TF-IDF word and two-word features. A random split would report macro F1 0.827; the time split gives 0.742, which is the figure a live inbox would see.

0%20%40%60%80%1,0003,00010,00030,000135,341Macro F1
More labelled complaints still help, with diminishing returns after 30,000.
TeamWords the model leans on
Bank accountsbank, chime, funds, savings, checking
Cardscard, comenity, synchrony, barclays, credit
Credit reportsequifax, experian, transunion, inquiries, xxxx xxxx
Debt collectiondebt, collection, collections, owe, medical
Money transferpaypal, venmo, coinbase, transfer, zelle
Mortgagesmortgage, escrow, home, modification, foreclosure
Personal loansloan, affirm, line of, finance, one main
Student loansnavient, mohela, student, loans, nelnet
Vehicle loanscar, vehicle, ally, auto, loan

Assumptions and limitations

  • US complaints and US product names; an Australian firm refits on a few thousand of its own labelled complaints, usually its last 6 to 12 months of help-desk tickets
  • Routing only; urgency, hardship and vulnerability flags are separate checks a person still makes
  • Accuracy per team ranges from 46% to 96%; small queues need more examples
  • Text is redacted before scoring; no personal details leave the help desk

What a complaints team would do next

  • Week 1: export 12 months of tickets with their final queue
  • Week 2: train, then set the bar on the last quarter for the accuracy the team accepts
  • Weeks 3 to 6: suggest the queue to the triage officer without moving tickets
  • Then: route automatically above the bar; review weekly misroutes

Technical depth

TF-IDF on words and two-word phrases with multinomial logistic regression, chosen for speed and for readable per-team words. Confidence is the top class probability; the 0.80 bar was set on the selection months.

Analysis code (Python)

Financial impact

About 182 hours a month of triage and re-routing removed at 1,500 complaints

Two savings: the triage officer no longer reads complaints routed automatically, and fewer complaints take a wrong hop compared with keyword rules.

InputValueSource
Complaints a month1,500 mid-sized lender, insurer or utility
Minutes to read and forward a complaint3
Routed automatically83%Measured
Wrong team: keyword rules43.8%Measured
Wrong team: model11.8%Measured
Minutes lost per wrong hop15 read, return, re-route, chase
Staff cost per hourA$55 loaded

Calculation

  1. Reading time saved62 h a month
  2. Wrong hops avoided480 a month
  3. Re-routing time saved120 h a month
  4. Annual valueA$120,285

Annual value of time savedA$120,285

Days recovered against the RG 271 limit and fewer AFCA escalations are not priced.

Implementation and rollout

Six weeks from ticket export to automatic routing above the bar

  1. Week 1

    Export

    12 months of tickets with their final queue from Zendesk, Freshdesk, Dynamics or Outlook.

    Complaints manager, Manaz

  2. Week 2

    Train and set the bar

    Train; set the bar on the last quarter for the accuracy the team accepts.

    Manaz, complaints manager

  3. Weeks 3 to 6

    Suggest mode

    Queue suggested to the triage officer without moving tickets.

    Triage officers

  4. Then

    Route above the bar

    Automatic routing above the bar; weekly misroute review; monthly retraining.

    Complaints manager

Tools

Zendesk, Freshdesk, Dynamics or a shared Outlook inbox, Redaction step before scoring, Python (scikit-learn) model, Weekly misroute report

Team

  • Complaints manager owns queues, bar and AFCA reporting
  • Triage officers handle held complaints and corrections
  • Manaz model, integration and monitoring

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Hardship or vulnerability missedRouting only; a person still checks urgency, hardship and vulnerability flags.
Small queues route poorlyAccuracy per team ranges from 46% to 96%; small queues get more examples or stay manual.
Personal information exposedText redacted before scoring; no personal details leave the help desk.
US product names do not transferRefit on 6 to 12 months of the firm’s own tickets.

Key takeaways

Key takeaways

  • Keyword rules misroute almost half of complaints on this data.
  • A confidence bar set on recent months gives a predictable accuracy for automatic routing.
  • Time splits matter: a random split would overstate macro F1 by 0.085.
  • Only the queue is chosen; nothing is answered automatically.