Manaz Afzal
Telecommunications

A watchlist of the broadband lines falling short of their plan at evening peak

Tested on 3,108 real connection-months of US speed tests from the FCC’s Measuring Broadband America programme.

86%of lines under the line stay under it next month
16%of next month’s cases pass today
49%of those caught in a tenth of passing lines
39%for the lines closest to the line

Executive summary

Last month’s speed tests already name 84% of next month’s under-delivering lines; a score on the passing lines catches 49% of those about to slip.

86%of failing lines still fail next month
16%of next month’s cases pass today
49%of those caught in a tenth of passing lines

Situation

Lines that fall short of their plan at evening peak reach the provider as complaints and credits.

Approach

A monthly watchlist from speed tests against the plan, tested on 3,108 real US connection-months.

Result

Failing lines stay failing; the score catches 49% of the slipping ones against 39%.

The problem

Fixed broadband in Australia is judged on busy-hour speed: the ACCC’s Measuring Broadband Australia programme reports how close each provider gets to plan speed between 7 and 11 pm on weeknights. Averages hide the lines that fall short. Network teams usually work from last month’s worst lines, and the lines that slip next month arrive as complaints and credits first.

I wanted to measure how well this month’s evening readings predict next month’s under-delivering lines, and whether a score can find the lines that pass today and fall short next month while there is time to fix them.

Team
Head of network operations, access network engineering, field service, customer care
Scale
Thousands to millions of fixed connections across DSL, cable and fibre
Systems
Line test and speed telemetry, network inventory, field service scheduling, CRM
Measure
Share of next month’s under-delivering lines on this month’s list; lines fixed before a complaint

What I built

A monthly watchlist with two parts. The first holds lines already under 80% of their advertised speed at evening peak, which go straight to engineering. The second ranks the lines that pass today by the chance they fall below the line next month, from the month’s speed tests: peak and off-peak medians against the plan, the slowest tenth of tests, the spread, the evening dip, last month’s result, technology and provider.

Speed testsHourly downloadsagainst the planThis monthPeak median, dip,slow testsWatchlistFailing now, andlikely to fail nextField workLines checked beforethe complaint
The list refreshes when a month of tests closes.
DSL23%23%Cable7%7%Fibre4%4%
DSL lines fall below the line far more often than cable or fibre, so the list leans on the copper estate. Data: FCC Measuring Broadband America, raw data 2023

Results

I tested on the FCC’s Measuring Broadband America raw data for January to July 2023: hourly download tests from volunteer households across US providers, joined to each unit’s advertised speed in the programme’s unit profile. 1,815 connections had at least ten evening-peak tests in a month and the next. The score was fitted on earlier months and tested on June and July, 3,108 connection-months, of which 15% were under 80% of the advertised speed at evening peak.

Last month’s result carries most of the signal: 86% of the lines under the line this month are still under it next month, and they make up 84% of next month’s cases. A score on the full set of readings matches that list and does not improve it, so the first part of the watchlist stays a rule.

The gain is in the other 16%: lines that pass this month and fail the next. Among the passing lines 2.8% slip each month.

Watchlist score49% caught, 14% hit rate49%Closest to the line39% caught, 11% hit rate39%
Checking the top tenth of passing lines by score catches 49% of the lines that slip, against 39% for the tenth closest to the 80% line. The test months hold few such lines, so the gap rests on few lines.
Lines checked among those passingScore: share caughtScore: hit rateClosest to the line: share caughtClosest to the line: hit rate
5%31%17%29%17%
10%49%14%39%11%
20%65%9%52%7%

On the passing lines the score separates those that slip with an AUC of 0.79, against 0.72 for distance from the line.

Data and method

Data
FCC Measuring Broadband America fixed raw data, January to July 2023, download throughput tests (curr_httpgetmt), joined to the unit profile from the Thirteenth Report for provider, technology and advertised download speed. The FCC publishes the programme data for public use without restrictions.
Target
Median of the month’s successful download tests between 7 and 11 pm local time under 80% of the advertised download speed, the following month
Population
Units with at least ten evening-peak tests in both months; March 2023 is missing from the download, so February to April pairs are skipped
Methods
Gradient-boosted classifiers on one month of unit readings plus the previous month’s result; fitted on earlier months and tested on June and July targets
Comparisons
This month’s result for the full list; distance from the 80% line for the passing lines

Limitations

How I would take it live

Financial impact

Lines fixed before the complaint

Base size and visit cost are assumptions; the catch rates are measured.

InputValueSource
Fixed lines50,000
Cost per truck rollA$180
Slipping lines a month2.8%Measured
Caught, score vs line49% vs 39%Measured

Calculation

  1. Slipping lines a month1,419
  2. Extra caught by the score151

151 more lines a month reviewed before a complaint

The early-warning gain comes from few test cases; a provider’s own base narrows it quickly.

Implementation and rollout

Two months from telemetry to a checked list

  1. Weeks 1 to 3

    Join

    Line telemetry joined to plan speeds.

    Network operations, Manaz

  2. Week 4

    Rank

    Failing lines to engineering; passing lines ranked.

    Access engineering, Manaz

  3. One month

    Check

    Top band checked and findings recorded.

    Field service

  4. After the month

    Adopt

    Complaints from listed lines tracked.

    Head of network operations

Tools

Line speed telemetry, Network inventory, Field service scheduling, Python, LightGBM

Team

  • Head of network operations owns the list
  • Access engineering diagnoses each line
  • Field service fixes on site
  • Manaz score and monthly refresh

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Panel dataProvider telemetry in production.
Few slip casesRange narrows on a full base.
Home Wi-FiEngineering review before a truck roll.
Plan changesPlan speeds read from billing.

Key takeaways

Key takeaways

  • Failing lines stay failing.
  • A sixth of next month’s cases pass today.
  • The score finds more of those than distance from the line.
  • The list starts the review, which decides the fix.