A watchlist of the broadband lines falling short of their plan at evening peak
Tested on 3,108 real connection-months of US speed tests from the FCC’s Measuring Broadband America programme.
Executive summary
Last month’s speed tests already name 84% of next month’s under-delivering lines; a score on the passing lines catches 49% of those about to slip.
Situation
Lines that fall short of their plan at evening peak reach the provider as complaints and credits.
Approach
A monthly watchlist from speed tests against the plan, tested on 3,108 real US connection-months.
Result
Failing lines stay failing; the score catches 49% of the slipping ones against 39%.
The problem
Fixed broadband in Australia is judged on busy-hour speed: the ACCC’s Measuring Broadband Australia programme reports how close each provider gets to plan speed between 7 and 11 pm on weeknights. Averages hide the lines that fall short. Network teams usually work from last month’s worst lines, and the lines that slip next month arrive as complaints and credits first.
I wanted to measure how well this month’s evening readings predict next month’s under-delivering lines, and whether a score can find the lines that pass today and fall short next month while there is time to fix them.
- Team
- Head of network operations, access network engineering, field service, customer care
- Scale
- Thousands to millions of fixed connections across DSL, cable and fibre
- Systems
- Line test and speed telemetry, network inventory, field service scheduling, CRM
- Measure
- Share of next month’s under-delivering lines on this month’s list; lines fixed before a complaint
What I built
A monthly watchlist with two parts. The first holds lines already under 80% of their advertised speed at evening peak, which go straight to engineering. The second ranks the lines that pass today by the chance they fall below the line next month, from the month’s speed tests: peak and off-peak medians against the plan, the slowest tenth of tests, the spread, the evening dip, last month’s result, technology and provider.
Results
I tested on the FCC’s Measuring Broadband America raw data for January to July 2023: hourly download tests from volunteer households across US providers, joined to each unit’s advertised speed in the programme’s unit profile. 1,815 connections had at least ten evening-peak tests in a month and the next. The score was fitted on earlier months and tested on June and July, 3,108 connection-months, of which 15% were under 80% of the advertised speed at evening peak.
Last month’s result carries most of the signal: 86% of the lines under the line this month are still under it next month, and they make up 84% of next month’s cases. A score on the full set of readings matches that list and does not improve it, so the first part of the watchlist stays a rule.
The gain is in the other 16%: lines that pass this month and fail the next. Among the passing lines 2.8% slip each month.
| Lines checked among those passing | Score: share caught | Score: hit rate | Closest to the line: share caught | Closest to the line: hit rate |
|---|---|---|---|---|
| 5% | 31% | 17% | 29% | 17% |
| 10% | 49% | 14% | 39% | 11% |
| 20% | 65% | 9% | 52% | 7% |
On the passing lines the score separates those that slip with an AUC of 0.79, against 0.72 for distance from the line.
Data and method
- Data
- FCC Measuring Broadband America fixed raw data, January to July 2023, download throughput tests (curr_httpgetmt), joined to the unit profile from the Thirteenth Report for provider, technology and advertised download speed. The FCC publishes the programme data for public use without restrictions.
- Target
- Median of the month’s successful download tests between 7 and 11 pm local time under 80% of the advertised download speed, the following month
- Population
- Units with at least ten evening-peak tests in both months; March 2023 is missing from the download, so February to April pairs are skipped
- Methods
- Gradient-boosted classifiers on one month of unit readings plus the previous month’s result; fitted on earlier months and tested on June and July targets
- Comparisons
- This month’s result for the full list; distance from the 80% line for the passing lines
Limitations
- The panel is volunteer households in the United States and its own test boxes, not a provider’s full line telemetry. On a live network the score adds line tests, modem counters and node load.
- Few passing lines slip in the two test months, so the early-warning gain carries a wide range. A provider’s base gives a tighter measure within weeks.
- Advertised speeds come from the programme’s profile, which can lag a plan change.
- A slow evening can come from home Wi-Fi or a neighbour-shared node; the list starts the engineering review, which decides the fix.
How I would take it live
- Join two months of line speed telemetry to plan speeds and technology
- Send the failing lines to engineering and rank the passing ones
- Check the top band for one month and record what was found
- Adopt with the share of complaints from lines already on the list tracked each month