The business problem
A mobile operator’s retention team has a fixed number of save calls and offers each week. Most teams pick who to call from simple triggers: a complaint, a contract coming off term, a drop in usage. Operators that rank every subscriber by predicted risk and match the offer to the reason report noticeably lower churn in the segments they treat, but only when the list is small enough for the team to work.
I measured how many of the following month’s leavers a ranked weekly list reaches at the size a retention team can call, against a complaint-and-usage trigger, and wrote the reasons behind each score into the CRM so the caller knows what to offer.
- Team
- 2 to 4 retention staff, part-time on outbound
- Capacity
- Roughly a fifth of the base contacted per quarter at most
- Systems
- Billing and usage records, HubSpot or Salesforce, a campaign tool such as Braze or Salesforce Marketing Cloud
- Constraint
- Privacy Act 1988 and the Telecommunications Consumer Protections Code; no automated decisions on price or service
What I built
A weekly scoring job reads call seconds and frequency, SMS count, distinct numbers called, call failures, complaints, tenure, tariff, charge band and account status. Each subscriber gets a risk score and the three strongest reasons ("call seconds down to the lowest fifth; two failed calls; complaint open"). Score and reasons go to a HubSpot contact property, and the top of the list becomes a task queue sized to the team.
Scope
- Connectors: HubSpot (contacts, properties, tasks), billing and usage exports, support-desk complaint records
- Weekly refresh, list capped to team capacity, reasons in plain language
- Outcome field on each task so saves and losses are recorded
- Monthly report: churn rate, list precision, value retained against a hold-out group
Results
| List | Subscribers | Churners reached | Churned customer value reached | Listed who left |
|---|---|---|---|---|
| Risk score, top 10% | 285 | 62.8% | 62.9% | 98.2% |
| Risk score, top 20% | 570 | 96.0% | 95.9% | 75.1% |
| Complaint, inactive, then low usage, top 20% | 570 | 72.9% | 66.0% | 57.0% |
| Lowest usage first, top 20% | 570 | 47.8% | 19.8% | 37.4% |
| Random, top 20% | 570 | 20.0% | 20.0% | 15.6% |
At the top 20% the score reaches 23% more of the churners than the rule. Resampling the subscribers a thousand times puts that gap between 20 and 27 percentage points (95% interval), so it is not an accident of this split. Separation this clean is unusual: the operator's own usage records carry a strong signal three months ahead, and a business with weaker data should expect less.
Where the risk sits
What it is worth
The dataset carries the operator's own customer value for every subscriber. The top-20% list reaches 96% of the value held by subscribers who left, against 66% for the rule. Retention calls commonly save 10% to 20% of the at-risk customers reached. At 15%, the score's list keeps about 14% of the value that would leave, and the rule's list 10%.
Data and method
- Source
- Iranian Churn Dataset, UCI Machine Learning Repository (Jafari-Marandi et al.), CC BY 4.0; records from an Iranian mobile operator
- Size
- 3,150 rows; 300 exact duplicates removed, 2,850 subscribers kept
- Window
- Usage aggregated over months 1 to 9; churn status at month 12, a three-month planning gap
- Churn rate
- 15.6%
A gradient-boosted classifier was scored out-of-fold with 5-fold stratified cross-validation, repeated over five shuffles (AUC 0.988 to 0.989). Raw age is left out in favour of the age group. The table shows how much AUC falls when each group of signals is removed and the model refitted.
| Signal group removed | Drop in AUC |
|---|---|
| Usage (seconds, calls, SMS, distinct numbers) | 0.020 |
| Tenure and age group | 0.014 |
| Complaints and call failures | 0.013 |
| Plan, charge band and status | 0.004 |
| Customer value | -0.000 |
Assumptions and limitations
- The data has no calendar dates, so the test is cross-validation within one cohort rather than a later period. In use the model is refitted monthly and checked on the following month
- The save rate is an industry range, not measured here. The first two months should hold out a random fifth of the list as a control
- One operator, prepaid-heavy, in one market. An Australian NBN or SaaS business will see different drivers (logins, seats, outages), so the model is refitted on its own history
- Usage aggregated over nine months hides the timing of a drop; a live system should add month-on-month change
Implementation
- Week 1: export 12 months of customers, usage, complaints, plan changes and cancellations from billing and HubSpot
- Week 2: refit, check against the last quarter, agree list size with the team
- Weeks 3 to 8: run the queue with a 20% random hold-out to measure saves
- Then: keep, resize or stop based on value retained against staff time
Technical depth
Out-of-fold scoring means every subscriber's score comes from a model trained without them. The hand rule ranks complainers first, then inactive accounts, then the lowest call seconds, which is the strongest ordering I could build from single fields. Group importance is measured by dropping each group, retraining and comparing out-of-fold AUC; usage and tenure carry most of the signal, and customer value adds nothing once usage is known.