Manaz Afzal
Telecommunications · Subscription businesses

A retention queue that ranks subscribers by risk of leaving

A weekly list for the retention team, written into HubSpot or Salesforce with the reasons behind each score. Tested on 2,850 subscribers of a mobile operator, with nine months of usage and the outcome three months later: the top fifth of the list holds 96% of the subscribers who left, against 73% for the best rule the team could run by hand.

96%of churners in the top 20%, against 73% for complaint, inactive, then low usage
75%of the top-20% list went on to leave, against 57% for the rule and 16% at random
0.988AUC out-of-fold, against 0.909 for the rule
3 monthsbetween the last usage reading and the churn outcome

Executive summary

A weekly retention list reaches 96% of the subscribers about to leave in a fifth of the base, against 73% for the best hand rule.

96%of churners in the top 20%, against 73% for complaint, inactive, then low usage
75%of the listed subscribers went on to leave, against 57% for the rule
+23 ptsgap at the top 20% (95% interval 20 to 27)

Situation

A small retention team can call about a fifth of the base a quarter. Calling complainers and quiet accounts first is sensible, and it still leaves a quarter of the leavers off the list.

Approach

Gradient boosting on nine months of usage, failures, complaints, tariff and status for 2,850 subscribers of a mobile operator, scored out-of-fold, with churn measured three months after the last reading.

Result

The top 20% holds 96% of churners against 73% for the rule; three in four people on the list are genuinely at risk. The data has no calendar dates, so a live run is checked month by month.

The business problem

A mobile operator’s retention team has a fixed number of save calls and offers each week. Most teams pick who to call from simple triggers: a complaint, a contract coming off term, a drop in usage. Operators that rank every subscriber by predicted risk and match the offer to the reason report noticeably lower churn in the segments they treat, but only when the list is small enough for the team to work.

I measured how many of the following month’s leavers a ranked weekly list reaches at the size a retention team can call, against a complaint-and-usage trigger, and wrote the reasons behind each score into the CRM so the caller knows what to offer.

Team
2 to 4 retention staff, part-time on outbound
Capacity
Roughly a fifth of the base contacted per quarter at most
Systems
Billing and usage records, HubSpot or Salesforce, a campaign tool such as Braze or Salesforce Marketing Cloud
Constraint
Privacy Act 1988 and the Telecommunications Consumer Protections Code; no automated decisions on price or service

What I built

A weekly scoring job reads call seconds and frequency, SMS count, distinct numbers called, call failures, complaints, tenure, tariff, charge band and account status. Each subscriber gets a risk score and the three strongest reasons ("call seconds down to the lowest fifth; two failed calls; complaint open"). Score and reasons go to a HubSpot contact property, and the top of the list becomes a task queue sized to the team.

Billing and usageSeconds, SMS, failures,tariff, statusWeekly scoreRisk and the top threereasons per subscriberHubSpotChurn-risk property;tasks for the top of listRetention callOffer chosen by theteam, not the modelOutcomeSaved, churned or nocontact; feeds the refit
Weekly workflow. The team decides the offer; the list only decides the order.

Scope

  • Connectors: HubSpot (contacts, properties, tasks), billing and usage exports, support-desk complaint records
  • Weekly refresh, list capped to team capacity, reasons in plain language
  • Outcome field on each task so saves and losses are recorded
  • Monthly report: churn rate, list precision, value retained against a hold-out group

Results

0%25%50%75%100%0%20%40%60%80%100%Best hand ruleLowest usageRandomRisk scoreTop 20%: 96% of churnersBest hand rule: 73%Share of the subscriber base on the retention list
Every subscriber was scored by a model that never saw them during training (5-fold, out-of-fold, averaged over five shuffles). Data: Iranian Churn Dataset, UCI Machine Learning Repository, CC BY 4.0, 2,850 subscribers
ListSubscribersChurners reachedChurned customer value reachedListed who left
Risk score, top 10%28562.8%62.9%98.2%
Risk score, top 20%57096.0%95.9%75.1%
Complaint, inactive, then low usage, top 20%57072.9%66.0%57.0%
Lowest usage first, top 20%57047.8%19.8%37.4%
Random, top 20%57020.0%20.0%15.6%
ABTop 10% of subscribers63%Top 20%96%Top 30%99%
At each list size the score reaches more of the churners than the best hand rule; the gap is widest at the list size a small team can work.

At the top 20% the score reaches 23% more of the churners than the rule. Resampling the subscribers a thousand times puts that gap between 20 and 27 percentage points (95% interval), so it is not an accident of this split. Separation this clean is unusual: the operator's own usage records carry a strong signal three months ahead, and a business with weaker data should expect less.

Where the risk sits

Usage fifth 1 (lowest)38%Usage fifth 221%Usage fifth 314%Usage fifth 46%Usage fifth 5 (highest)0%All subscribers 15.6%
Churn falls steadily with usage: more than a third of the lowest-usage fifth left, and nobody in the highest fifth did.
Filed a complaint230 subscribers83%Inactive status684 subscribers48%Pay-as-you-go tariff2,621 subscribers17%No complaint2,620 subscribers10%Active status2,166 subscribers6%Contract tariff229 subscribers3%All subscribers 15.6%
A complaint is the strongest single warning: 83% of subscribers who complained left. Contract tariffs churn at 3%.
Score tenth 1098%Score tenth 952%Score tenth 85%Score tenth 71%Score tenth 61%Score tenth 50%Score tenth 40%Score tenth 30%Score tenth 20%Score tenth 10%Base rate
The score has a sharp edge: the top tenth churns at 98%, and the bottom half almost never.

What it is worth

The dataset carries the operator's own customer value for every subscriber. The top-20% list reaches 96% of the value held by subscribers who left, against 66% for the rule. Retention calls commonly save 10% to 20% of the at-risk customers reached. At 15%, the score's list keeps about 14% of the value that would leave, and the rule's list 10%.

For a business with 7,000 subscribers at similar churn, the same list would be about 1,400 people a quarter, or 20 to 25 calls a working day for one person, and about three in four of them would be genuinely at risk.

Data and method

Source
Iranian Churn Dataset, UCI Machine Learning Repository (Jafari-Marandi et al.), CC BY 4.0; records from an Iranian mobile operator
Size
3,150 rows; 300 exact duplicates removed, 2,850 subscribers kept
Window
Usage aggregated over months 1 to 9; churn status at month 12, a three-month planning gap
Churn rate
15.6%

A gradient-boosted classifier was scored out-of-fold with 5-fold stratified cross-validation, repeated over five shuffles (AUC 0.988 to 0.989). Raw age is left out in favour of the age group. The table shows how much AUC falls when each group of signals is removed and the model refitted.

Signal group removedDrop in AUC
Usage (seconds, calls, SMS, distinct numbers)0.020
Tenure and age group0.014
Complaints and call failures0.013
Plan, charge band and status0.004
Customer value-0.000

Assumptions and limitations

  • The data has no calendar dates, so the test is cross-validation within one cohort rather than a later period. In use the model is refitted monthly and checked on the following month
  • The save rate is an industry range, not measured here. The first two months should hold out a random fifth of the list as a control
  • One operator, prepaid-heavy, in one market. An Australian NBN or SaaS business will see different drivers (logins, seats, outages), so the model is refitted on its own history
  • Usage aggregated over nine months hides the timing of a drop; a live system should add month-on-month change

Implementation

  • Week 1: export 12 months of customers, usage, complaints, plan changes and cancellations from billing and HubSpot
  • Week 2: refit, check against the last quarter, agree list size with the team
  • Weeks 3 to 8: run the queue with a 20% random hold-out to measure saves
  • Then: keep, resize or stop based on value retained against staff time

Technical depth

Out-of-fold scoring means every subscriber's score comes from a model trained without them. The hand rule ranks complainers first, then inactive accounts, then the lowest call seconds, which is the strongest ordering I could build from single fields. Group importance is measured by dropping each group, retraining and comparing out-of-fold AUC; usage and tenure carry most of the signal, and customer value adds nothing once usage is known.

Analysis code (Python, LightGBM)

Financial impact

About A$81,966 a year in retained revenue for a 7,000-subscriber base

Value comes from the extra at-risk subscribers the team reaches with the same call capacity, multiplied by the share a call keeps.

InputValueSource
Subscribers7,000 regional ISP, MVNO or subscription business
Quarterly churn15.6%Measured measured in this data
Churners reached, top 20%: score96%Measured
Churners reached, top 20%: rule73%Measured
Share of reached churners a call keeps15% industry range 10% to 20%
Monthly revenue per subscriberA$45

Calculation

  1. Extra churners reached a quarter253
  2. Extra subscribers kept a year152
  3. Their next twelve months of revenueA$81,966

Annual revenue retained over the hand ruleA$81,966

The save rate is not measured in this data; the first two months run with a random hold-out to measure it.

Implementation and rollout

Six weeks from billing export to a retention queue the team works every Monday

  1. Week 1

    Extract

    Export twelve months of usage, complaints, plan changes and cancellations from billing and HubSpot.

    Billing lead, Manaz

  2. Week 2

    Refit and backtest

    Refit on the business’s own history and test on the latest quarter against the current rule.

    Manaz

  3. Weeks 3 to 6

    Run with a hold-out

    Queue in HubSpot with reasons; a random fifth of the list is held back to measure saves.

    Retention team

  4. Then

    Decide

    Keep, resize or stop on revenue retained against staff time.

    Customer lead, Manaz

Tools

Billing and usage exports, HubSpot or Salesforce properties and tasks, Python LightGBM, scored weekly, Looker Studio or Power BI for the monthly report

Team

  • Customer or retention lead owns offers and list size
  • Retention staff work the queue and record outcomes
  • Manaz pipeline, model, reasons and monthly checks

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Offer fatigue: the same people called every weekCool-off period after each contact; list excludes recent calls.
Model learns the operator’s own past actionsHold-out group and monthly refit on outcomes, not on who was called.
Usage feed arrives lateScore runs on the last complete week and flags missing data.
Score read as a reason to change price or serviceScore only orders the list; offers stay with the team, as the TCP Code expects.

Key takeaways

Key takeaways

  • A complaint is the strongest single warning: 83% of complainers left.
  • Usage carries most of the signal; customer value adds nothing once usage is known.
  • The gap over the best rule holds under resampling (20 to 27 points).
  • Data this clean is unusual; a business with weaker records should expect a smaller gain and test it.