Manaz Afzal

Insurance

The complaints that will be upheld, at the top of the queue on the day they arrive

An intake score for an insurer’s complaints team, tested on 64,462 real complaints to a state insurance regulator against working the queue in arrival order.

45%of confirmed complaints in the top 30% of the queue; reason rule 36%
55%confirmed in the top tenth, against 28% overall
0.75AUC for complaints that end in an extra payment

Executive summary

An intake score puts 45% of regulator-confirmed complaints in the top 30% of an insurer’s queue, against 36% for a reason-based rule, tested on 64,462 later complaints.

45%of confirmed complaints in the top 30% of the queue; rule 36%
55%confirmed in the top tenth, against 28% overall
0.75AUC for complaints ending in an extra payment

Situation

An insurer’s complaints team works its queue in arrival order, so the complaints most likely to be upheld wait behind routine ones.

Approach

An intake score from reason, line of business, complainant role and the insurer’s record, fitted on 115,785 real complaints and tested on 64,462 later ones.

Result

The top 30% of the queue holds 45% of confirmed complaints; the top tenth is confirmed 55% of the time.

The problem

An insurer’s internal dispute resolution team works complaints in arrival order against RG 271 time limits. Some of those complaints will end up upheld by AFCA or the regulator, and finding them on day one lets a senior case manager fix them before they escalate. AFCA’s Datacube and ASIC’s IDR dashboard show how outcomes vary by product and issue.

I measured how many of the complaints that would be upheld an intake score puts at the top of the queue on the day they arrive, against working in arrival order.

Team
Head of complaints, internal dispute resolution handlers, claims and underwriting leads
Scale
Thousands of complaints a year across motor, home, health and life
Systems
Salesforce Service Cloud or Dynamics 365, Guidewire or Duck Creek for claims
Measure
Confirmed complaints reached in the first review pass; days to senior review

What I built

An intake score that rates each new complaint for the chance it will be confirmed against the insurer, using only fields captured on arrival. The queue is sorted by that score, so a senior handler opens the highest-risk complaints first and the routine ones flow to the general queue. A second score rates the chance the customer ends up receiving more money, which flags complaints worth settling early.

Complaint arrivesPortal, emailor regulator referralIntake scoreReason, line, role,insurer recordSenior queueHighest-riskcomplaints firstGeneral queueRoutine complaintsin arrival order
Handlers see the reasons behind each score, and every closed complaint feeds the next refit.

Results

I tested on real complaints made to the Texas Department of Insurance: 180,247 closed complaint lines against insurers from 2018 to 2026, each with the regulator’s finding. The score was fitted on complaints received before 2024 and tested on the 64,462 received from January 2024 onward.

0%20%40%60%0%10%20%30%40%50%Intake scoreReason-based ruleArrival orderTop 30% of the queue:45% of confirmed complaintsShare of new complaints given senior review first
Reviewing the top 30% of the queue first reaches 45% of the complaints the regulator went on to confirm, against 36% for a rule that ranks by the complaint reason alone. Data: Texas Department of Insurance, insurance complaints, data.texas.gov

In the top tenth of the queue, 55% of complaints were confirmed, almost twice the 28% average. The score separates confirmed from unconfirmed complaints with an AUC of 0.66; the reason-based rule manages 0.58 and the insurer’s past record alone 0.58.

Share of queue reviewed firstConfirmed complaints found: intake scoreReason-based rule
10%19%14%
20%33%25%
30%45%36%
40%56%47%
50%66%59%
Score decile 10predicted 40%55%Score decile 9predicted 26%41%Score decile 8predicted 20%35%Score decile 7predicted 17%29%Score decile 6predicted 15%28%Score decile 5predicted 13%25%Score decile 4predicted 12%24%Score decile 3predicted 10%18%Score decile 2predicted 8%17%Score decile 1predicted 5%14%
The score is ordered: each tenth of the queue is confirmed less often than the one above it.

Which customers get more money

The second score, for complaints that end with the customer receiving additional money, reaches an AUC of 0.75. 24% of confirmed complaints ended in an extra payment, against 9% of the rest. These are the complaints where an early, fair offer shortens the dispute.

When the base rate moves

The share of complaints confirmed against insurers sat between 12% and 15% from 2018 to 2022, reached 19% in 2023 and has run at about 28% since 2024. A model fitted on the earlier years under-predicts the level, while its ranking still holds. I check the level each month and refit the calibration when it drifts, so the queue order and the forecast workload both stay right.

201828,192 complaint lines15%201920,393 complaint lines15%202013,434 complaint lines15%202117,049 complaint lines12%202216,267 complaint lines15%202320,450 complaint lines19%202422,340 complaint lines29%202521,789 complaint lines28%202620,333 complaint lines29%
The jump in 2024 is a change in findings, not in the score. The AUC of 0.66 is measured entirely on complaints received after that change.
Accident and Health12,352 lines42%Life & Annuity3,673 lines31%Automobile29,287 lines26%Homeowners14,873 lines25%Miscellaneous2,231 lines22%Liability1,536 lines19%
Health complaints are confirmed most often, which is why line of business carries weight in the score.

Data and method

Data
Texas Department of Insurance complaints against insurance companies, published on data.texas.gov: 180,247 closed complaint lines received 2018-01-02 to 2026-11-06, with reason, line of business, complainant role, the insurer and the regulator’s finding. No complainant is identified in the data or here.
Target
Complaint confirmed by the regulator; separately, additional money received by the complainant
Split
Fitted on complaints received before January 2024; tested on January 2024 to November 2026
Methods
Logistic regression on complaint reasons, line of business, coverage level, complainant role and type, and a smoothed confirmed rate for the insurer from the fitting years only; compared with arrival order and a reason-based rule
Measures
Confirmed complaints reached by share of queue reviewed; AUC; calibration by decile

Limitations

  • A regulator’s complaint file stands in for an insurer’s internal register. An insurer’s own data adds policy, claim and contact history, which should strengthen the score.
  • The finding is the regulator’s judgement, and its standard shifted in 2024; the monthly level check exists for that reason.
  • The score orders the queue; it does not decide outcomes. Every complaint is still handled against the insurer’s internal dispute resolution standard and response timeframes.
  • Insurers are not named here.

How I would take it live

  • Export two years of closed complaints with outcomes from the complaints register
  • Fit the intake score and the payment score, and agree the senior-queue share with the head of complaints
  • Run in shadow for a month: handlers work as normal while the score is recorded and compared with outcomes
  • Sort the live queue by score in Salesforce or Dynamics, with a monthly check of calibration and outcomes

Financial impact

Upheld complaints reach a senior handler weeks sooner

The gain is earlier senior review of the complaints most likely to be upheld. Complaint numbers, the wait in arrival order and the cost per handler hour are assumptions.

InputValueSource
Complaints a year6,000 mid-size general insurer
Typical wait for senior review in arrival order21 days
Share of queue given senior review first30%
Confirmed complaints in that share45%Measured test on 2024 to 2026
Loaded handler cost an hourA$140

Calculation

  1. Likely-upheld complaints reviewed in the first pass777 a year
  2. Against arrival order513 a year

265 more likely-upheld complaints a year reach senior review first

Fewer escalations to the external dispute scheme and earlier fair offers are further gains and are not counted here.

Implementation and rollout

Four weeks from export to a scored queue

  1. Week 1

    Export

    Two years of closed complaints with outcomes from the complaints register.

    Head of complaints, Manaz

  2. Week 2

    Fit and agree

    Intake and payment scores fitted; senior-queue share agreed.

    Complaints leads, Manaz

  3. Week 3

    Shadow month begins

    Handlers work as normal; scores recorded and compared with outcomes.

    Handlers

  4. Week 4 onward

    Live queue

    Queue sorted by score in the case system; monthly calibration check.

    Complaints team, Manaz on support

Tools

Salesforce Service Cloud or Dynamics 365, Guidewire or Duck Creek claims data, Python, scikit-learn

Team

  • Head of complaints owns the queue rules
  • Senior handlers work the high-score queue
  • Claims and underwriting leads act on root causes
  • Manaz scores, integration and monthly review

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
The regulator’s standard shiftsMonthly level check and recalibration; ranking is monitored separately.
Scores read as verdictsThe score orders the queue only; every complaint is handled on its merits.
Response timeframesEvery complaint keeps its statutory clock regardless of score.
Bias against a customer groupScore audited by complainant type and line each quarter.

Key takeaways

Key takeaways

  • Upheld complaints can be recognised on the day they arrive.
  • The top 30% of the queue holds 45% of them.
  • A second score flags complaints that end with extra payment, which are worth settling early.
  • Calibration is watched monthly because findings shift.