Fraud reports to the front of the queue, and every message to the right team
A triage step for a bank's chat and email desk, tested on 3,080 real customer messages across 77 request types.
Executive summary
A triage step in front of a bank’s help desk sends 97.1% of messages to the right queue, against 66% for keyword rules, and flags 97.5% of fraud and lost-card reports on arrival.
Situation
Every chat and email lands in one queue. Agents read messages only to forward them, and a stolen card waits behind exchange-rate questions. Keyword rules misroute a third and miss most fraud, because customers write “I don’t remember buying this”.
Approach
A text model scores 77 request types on the BANKING77 data (10,003 training and 3,080 test messages), sums them into eight queues, and raises an urgency flag at a low threshold so very few fraud reports are missed.
Result
Queue accuracy 97.1% (95% CI 96.5% to 97.8%); intent accuracy 91.4%. At a 0.80 bar 87% of messages are routed automatically at 99.6% accuracy; nothing is answered automatically.
The business problem
A digital bank’s support desk takes chat and email about cards, transfers, top-ups and identity checks, with lost cards and fraud mixed in among routine questions. As the team grows, tagging by hand drifts: agents pick different contact reasons for the same problem, and messages sit in the general queue until someone with the right skill picks them up. Banks and fintechs that route on predicted intent report that a large share of tickets can be placed without an agent reading them first, with fraud reports pulled forward.
I set out to measure how accurately a text model can send each message to the team that acts on it, how many it can route with high confidence, and how reliably it raises fraud and lost-card reports on arrival.
- Team
- A first-line support desk of 5 to 40 agents, with specialist queues behind it
- Scale
- Thousands of short chat and email messages a week
- Systems
- Zendesk, Intercom or Freshdesk, the core banking CRM, card management system
- Constraint
- Fraud reports have to reach the right team within minutes; ePayments Code and scheme rules
What was built
A triage step in front of the help desk. It reads each message, scores all 77 request types, adds them up into eight queues and raises an urgency flag for anything that looks like fraud or a lost card. Clear messages go straight to their queue. Messages below the confidence bar wait for an agent, with the top two queues suggested. Nothing is answered automatically.
The eight queues
| Queue | What it handles |
|---|---|
| Fraud and security | Lost or stolen cards and phones, compromised cards, payments and withdrawals the customer does not recognise |
| Disputes and refunds | Refunds, double charges, reverted payments, wrong amounts and wrong exchange rates |
| Cards | Ordering, delivery, activation, PINs, declines, virtual and disposable cards, wallets |
| Transfers and payments | Pending, failed, cancelled and incoming transfers, beneficiaries |
| Top-ups | Top-ups by card, transfer and cash, limits, failures and reversals |
| Cash and ATMs | ATM support, declined and pending withdrawals, withdrawal charges |
| Fees and FX | Fees, exchange rates, supported currencies and countries |
| Identity and account | Identity checks, source of funds, personal details, passcodes, closing an account |
Results
| Method | Right queue | 95% range | Macro F1 |
|---|---|---|---|
| Send everything to Cards | 27.3% | ||
| Keyword rules | 66.3% | 64.7% to 68.0% | 0.644 |
| Text model | 97.1% | 96.5% to 97.8% | 0.967 |
The urgency flag
| Urgency flag | Urgent caught | Flags that were urgent | Messages flagged |
|---|---|---|---|
| Keyword rules | 37.5% | 94.6% | 111 |
| Model, threshold 0.2 | 97.5% | 94.5% | 289 |
| Model, threshold 0.5 | 91.4% | 97.7% | 262 |
Where it goes wrong
Most mistakes sit between neighbouring queues: a top-up paid by bank transfer, or a refund for a cash withdrawal. These are the pairs a supervisor should review weekly.
| Actual queue | Sent to | Messages |
|---|---|---|
| Top-ups | Transfers and payments | 7 |
| Disputes and refunds | Cash and ATMs | 5 |
| Fees and FX | Disputes and refunds | 4 |
| Disputes and refunds | Fraud and security | 4 |
| Transfers and payments | Disputes and refunds | 4 |
| Top-ups | Cards | 4 |
Confident mistakes
The five wrong answers the model was most sure about. Several are reasonable readings of an ambiguous message.
| Message | Correct queue | Model sent to | Confidence |
|---|---|---|---|
| How do I top up my card? | Transfers and payments | Top-ups | 0.99 |
| I was double charged, and the second charge is showing as "pending". How long will it be before I get my money back once the second charge has been refunded? | Transfers and payments | Disputes and refunds | 0.98 |
| My cash withdrawal was partly declined | Disputes and refunds | Cash and ATMs | 0.97 |
| Will declined funds I tried to withdraw be returned to me? | Disputes and refunds | Cash and ATMs | 0.93 |
| I cannot find my credit card. | Fraud and security | Top-ups | 0.88 |
Data and method
- Data
- BANKING77 (Casanueva et al., 2020, PolyAI), CC BY 4.0: 13,083 customer queries to an online bank, each labelled with one of 77 request types
- Split
- The published split: 10,003 for training, 3,080 for testing. Nothing from the test set was used to tune the model or the bars
- Queues
- The 77 types were grouped into eight queues by me, by which team would act on them; the mapping is in the code
- Model
- Word and character n-gram TF-IDF with multinomial logistic regression; queue score is the sum of its request-type probabilities
- Baselines
- Keyword rules written per queue, and sending everything to the largest queue (Cards)
- Ranges
- 95% bootstrap ranges over test messages, 1,000 resamples
Limitations
- The test set has 40 messages per request type, so queues are evenly sized. A real inbox is uneven, and accuracy should be measured again on the bank's own mix.
- Messages are single questions in English with a median of 9 words. Long emails with several requests need splitting or a multi-label model.
- The queue design and keyword rules are mine. A team with different queues would see different numbers, though the gap between rules and the model is large enough to survive that.
- The urgency flag is measured on request types, not on time to action. Its value depends on the desk acting on the flag.
- No personal information is in the data; a live inbox would need redaction before scoring.
What a support team would do next
- Week 1: export three months of messages with the queue each one ended up in
- Week 2: map their request types to queues and train; set both bars on the most recent month
- Weeks 3 to 6: show the suggested queue and urgency flag to agents without moving anything
- Then: route automatically above the bar, keep the urgency flag on everything, review the confusion pairs weekly
Technical depth
Word unigrams and bigrams plus character 3 to 5-grams (TF-IDF, sublinear term frequency) feed a multinomial logistic regression with C = 20. The urgency score is the summed probability of the seven urgent request types, so it works even when the queue is unclear. Bars were fixed before looking at test results.