Manaz Afzal
Banking & Lending · Inbox and ticket triage

Fraud reports to the front of the queue, and every message to the right team

A triage step for a bank's chat and email desk, tested on 3,080 real customer messages across 77 request types.

97.1%sent to the right queue; keyword rules 66%
97.5%of urgent messages flagged on arrival
87%routed with no agent, 99.6% right
400held for an agent, with a suggestion

Executive summary

A triage step in front of a bank’s help desk sends 97.1% of messages to the right queue, against 66% for keyword rules, and flags 97.5% of fraud and lost-card reports on arrival.

97.1%of 3,080 test messages sent to the right queue (keyword rules 66.3%)
97.5%of urgent fraud and lost-card messages flagged, against 37.5% for rules
87%routed with no agent at 99.6% accuracy; 400 held with a suggestion

Situation

Every chat and email lands in one queue. Agents read messages only to forward them, and a stolen card waits behind exchange-rate questions. Keyword rules misroute a third and miss most fraud, because customers write “I don’t remember buying this”.

Approach

A text model scores 77 request types on the BANKING77 data (10,003 training and 3,080 test messages), sums them into eight queues, and raises an urgency flag at a low threshold so very few fraud reports are missed.

Result

Queue accuracy 97.1% (95% CI 96.5% to 97.8%); intent accuracy 91.4%. At a 0.80 bar 87% of messages are routed automatically at 99.6% accuracy; nothing is answered automatically.

The business problem

A digital bank’s support desk takes chat and email about cards, transfers, top-ups and identity checks, with lost cards and fraud mixed in among routine questions. As the team grows, tagging by hand drifts: agents pick different contact reasons for the same problem, and messages sit in the general queue until someone with the right skill picks them up. Banks and fintechs that route on predicted intent report that a large share of tickets can be placed without an agent reading them first, with fraud reports pulled forward.

I set out to measure how accurately a text model can send each message to the team that acts on it, how many it can route with high confidence, and how reliably it raises fraud and lost-card reports on arrival.

Team
A first-line support desk of 5 to 40 agents, with specialist queues behind it
Scale
Thousands of short chat and email messages a week
Systems
Zendesk, Intercom or Freshdesk, the core banking CRM, card management system
Constraint
Fraud reports have to reach the right team within minutes; ePayments Code and scheme rules

What was built

A triage step in front of the help desk. It reads each message, scores all 77 request types, adds them up into eight queues and raises an urgency flag for anything that looks like fraud or a lost card. Clear messages go straight to their queue. Messages below the confidence bar wait for an agent, with the top two queues suggested. Nothing is answered automatically.

MessageChat or email,free textScore77 request types,summed to 8 queuesUrgent checkFraud, lost card,unrecognised paymentRouteConfidence 0.80+:to the queue (87%)Agent picksBelow 0.80 (13%),top two suggestedLearnAgent correctionssaved for retraining
Where the triage step sits. Urgent messages are flagged at a low threshold so very few are missed, whatever their queue confidence.

The eight queues

QueueWhat it handles
Fraud and securityLost or stolen cards and phones, compromised cards, payments and withdrawals the customer does not recognise
Disputes and refundsRefunds, double charges, reverted payments, wrong amounts and wrong exchange rates
CardsOrdering, delivery, activation, PINs, declines, virtual and disposable cards, wallets
Transfers and paymentsPending, failed, cancelled and incoming transfers, beneficiaries
Top-upsTop-ups by card, transfer and cash, limits, failures and reversals
Cash and ATMsATM support, declined and pending withdrawals, withdrawal charges
Fees and FXFees, exchange rates, supported currencies and countries
Identity and accountIdentity checks, source of funds, personal details, passcodes, closing an account

Results

Keyword rulesModelIdentity and account100%Cards99%Fraud and security97%Top-ups97%Fees and FX97%Transfers and payments96%Disputes and refunds94%Cash and ATMs94%
The model sends 97.1% of messages to the right queue against 66.3% for keyword rules. Rules do worst on fraud (46%) and disputes, where customers describe the problem instead of naming it. Data: BANKING77, test set of 3,080 messages
MethodRight queue95% rangeMacro F1
Send everything to Cards27.3%
Keyword rules66.3%64.7% to 68.0%0.644
Text model97.1%96.5% to 97.8%0.967
50%60%70%80%90%100%0.30.40.50.60.70.80.9Accuracy of auto-routed messagesShare auto-routed0.80: 87% routed, 99.6% rightQueue confidence bar
At a 0.80 bar, 87% of messages are routed without an agent and 99.6% of those are right. The 400 held messages are the hard ones: the model's first choice is right for 80% of them, so the suggestion still saves the agent a step.

The urgency flag

60%70%80%90%100%0.10.20.30.40.50.60.7Flags that were urgentUrgent messages caught0.2: 97.5% caughtUrgency threshold
A low threshold of 0.2 catches 97.5% of the 280 urgent messages, and 94% of the flags are real. Keyword rules catch 38%.
Urgency flagUrgent caughtFlags that were urgentMessages flagged
Keyword rules37.5%94.6%111
Model, threshold 0.297.5%94.5%289
Model, threshold 0.591.4%97.7%262
In practice: 98% of fraud and lost-card messages jump the queue on arrival. With keyword rules, 62% of them would have waited in the general queue.

Where it goes wrong

Most mistakes sit between neighbouring queues: a top-up paid by bank transfer, or a refund for a cash withdrawal. These are the pairs a supervisor should review weekly.

Actual queueSent toMessages
Top-upsTransfers and payments7
Disputes and refundsCash and ATMs5
Fees and FXDisputes and refunds4
Disputes and refundsFraud and security4
Transfers and paymentsDisputes and refunds4
Top-upsCards4

Confident mistakes

The five wrong answers the model was most sure about. Several are reasonable readings of an ambiguous message.

MessageCorrect queueModel sent toConfidence
How do I top up my card?Transfers and paymentsTop-ups0.99
I was double charged, and the second charge is showing as "pending". How long will it be before I get my money back once the second charge has been refunded?Transfers and paymentsDisputes and refunds0.98
My cash withdrawal was partly declinedDisputes and refundsCash and ATMs0.97
Will declined funds I tried to withdraw be returned to me?Disputes and refundsCash and ATMs0.93
I cannot find my credit card.Fraud and securityTop-ups0.88

Data and method

Data
BANKING77 (Casanueva et al., 2020, PolyAI), CC BY 4.0: 13,083 customer queries to an online bank, each labelled with one of 77 request types
Split
The published split: 10,003 for training, 3,080 for testing. Nothing from the test set was used to tune the model or the bars
Queues
The 77 types were grouped into eight queues by me, by which team would act on them; the mapping is in the code
Model
Word and character n-gram TF-IDF with multinomial logistic regression; queue score is the sum of its request-type probabilities
Baselines
Keyword rules written per queue, and sending everything to the largest queue (Cards)
Ranges
95% bootstrap ranges over test messages, 1,000 resamples

Limitations

What a support team would do next

Technical depth

Word unigrams and bigrams plus character 3 to 5-grams (TF-IDF, sublinear term frequency) feed a multinomial logistic regression with C = 20. The urgency score is the summed probability of the seven urgent request types, so it works even when the queue is unclear. Bars were fixed before looking at test results.

Analysis code (Python)

Financial impact

About 72 agent hours a week no longer spent forwarding messages

Counted here is only the time first-line agents spend reading a message to forward it. Faster response to fraud and fewer misroutes are real but not priced.

InputValueSource
Messages a week5,000 mid-sized digital bank or card issuer
Minutes to read and forward a message1
Share routed automatically at the bar87%Measured
Accuracy of automatic routing99.6%Measured
Agent cost per hourA$45 loaded

Calculation

  1. Messages routed with no agent4,350 a week
  2. Agent hours saved a week72
  3. Annual valueA$169,650

Annual value of agent timeA$169,650

The ePayments Code and scheme rules make speed on fraud reports a compliance matter as well as a cost one.

Implementation and rollout

Seven weeks to automatic routing above the bar, with fraud flags live from week four

  1. Weeks 1 to 2

    Export and label

    Six to twelve months of messages with their final queue from Zendesk, Intercom or Service Cloud; map to the bank’s own queues.

    Support operations lead, Manaz

  2. Week 3

    Train and set bars

    Train, set the routing bar and the low urgency threshold with the fraud team.

    Manaz, fraud team lead

  3. Weeks 4 to 6

    Suggest mode

    Queue and urgency suggestions shown to agents; fraud flags go live; overrides logged.

    First-line agents

  4. Week 7 onward

    Route above the bar

    Messages above the bar routed automatically; monthly retraining on agent corrections.

    Support operations lead

Tools

Zendesk, Intercom, Salesforce Service Cloud or Freshdesk, Python text model behind a webhook, Help-desk triggers and views, Weekly accuracy dashboard

Team

  • Support operations lead owns queues and bars
  • Fraud team lead owns the urgency threshold
  • Manaz model, integration and monitoring

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
A fraud report is missedUrgency flag set at a low threshold (catches 97.5%); every flagged message reaches the fraud queue first.
New products create unseen request typesLow-confidence messages are held for an agent; monthly retraining adds new types.
Customer data handled outside the bankModel hosted in the bank’s cloud tenancy; no external model API.
Agents stop checking held suggestionsShow top two queues and confidence; track override rate.

Key takeaways

Key takeaways

  • Keyword rules misroute a third of messages and miss most fraud reports on this data.
  • Scoring 77 intents and summing to queues gives 97% queue accuracy with a simple, auditable model.
  • A low urgency threshold trades a few false flags for catching 97.5% of fraud and lost-card reports.
  • A confidence bar decides what is automated: 87% routed at 99.6%, the rest held for an agent.