The business problem
Under ASIC’s RG 271 a lender has to acknowledge a complaint within a business day and give a written response within 30 calendar days, and systemic issues have to be spotted early. In most complaints teams the first step is manual: someone reads the email or form, picks a product and issue category, and passes it to the team that owns it. When that first read is wrong, the complaint bounces between teams and the 30 days keep running.
I built a routing step that reads each complaint as the customer wrote it, sends it to the owning team, and holds anything it is unsure of for a complaints officer, then measured how often it gets the first hand-off right against the keyword rules many teams start with.
- Team
- One or two triage officers; five to ten specialist queues
- Scale
- Hundreds to thousands of complaints a month
- Systems
- Outlook or a shared complaints inbox, Salesforce Service Cloud, Dynamics 365 or a complaints system such as Complaints Pro
- Constraint
- Personal information in free text; Privacy Act; AFCA reporting
What was built
A triage step on the shared inbox. It reads each redacted complaint, scores all nine teams at once, sends the clear ones straight to a queue and leaves the rest for a person. Every correction is saved and the model is retrained monthly.
Results
| Method | Accuracy | Macro F1 | Balanced accuracy |
|---|---|---|---|
| Keyword rules | 56.2% | 0.415 | 44.3% |
| Send all to the biggest team | 70.5% | 0.092 | 11.1% |
| Text model | 88.2% | 0.742 (0.737–0.747) | 71.3% |
Data and method
- Source
- US Consumer Financial Protection Bureau, Consumer Complaint Database (public, narratives redacted by the CFPB)
- Window
- 665,936 complaints with text, July 2022 to December 2023
- Split
- Train to June 2023 (135,341 used), choose the bar on July to September, test once on 111,466 complaints from October to December
- Teams
- CFPB products mapped to nine queues; product renames in 2023 merged
Logistic regression on TF-IDF word and two-word features. A random split would report macro F1 0.827; the time split gives 0.742, which is the figure a live inbox would see.
| Team | Words the model leans on |
|---|---|
| Bank accounts | bank, chime, funds, savings, checking |
| Cards | card, comenity, synchrony, barclays, credit |
| Credit reports | equifax, experian, transunion, inquiries, xxxx xxxx |
| Debt collection | debt, collection, collections, owe, medical |
| Money transfer | paypal, venmo, coinbase, transfer, zelle |
| Mortgages | mortgage, escrow, home, modification, foreclosure |
| Personal loans | loan, affirm, line of, finance, one main |
| Student loans | navient, mohela, student, loans, nelnet |
| Vehicle loans | car, vehicle, ally, auto, loan |
Assumptions and limitations
- US complaints and US product names; an Australian firm refits on a few thousand of its own labelled complaints, usually its last 6 to 12 months of help-desk tickets
- Routing only; urgency, hardship and vulnerability flags are separate checks a person still makes
- Accuracy per team ranges from 46% to 96%; small queues need more examples
- Text is redacted before scoring; no personal details leave the help desk
What a complaints team would do next
- Week 1: export 12 months of tickets with their final queue
- Week 2: train, then set the bar on the last quarter for the accuracy the team accepts
- Weeks 3 to 6: suggest the queue to the triage officer without moving tickets
- Then: route automatically above the bar; review weekly misroutes
Technical depth
TF-IDF on words and two-word phrases with multinomial logistic regression, chosen for speed and for readable per-team words. Confidence is the top class probability; the 0.80 bar was set on the selection months.