Manaz Afzal

SaaS & Tech · Inbox and ticket triage

Bugs to engineering, questions to support, before anyone reads the issue

A labelling step for a product's public issue tracker, tested on 28,464 later GitHub issues.

85.1%labelled correctly; title rules 62%
19%of questions and docs issues reach engineering; rules 57%
0.74macro F1 across four types; rules 0.42

Executive summary

A labelling step on a public issue tracker cuts the share of questions and docs issues reaching engineering from 57% to 19%, with 85.1% of issues labelled correctly.

85.1%of later issues labelled correctly, against 62.3% for title and template rules
0.74macro F1 across four types; rules 0.42
71%labelled automatically at 93.3% accuracy

Situation

Customers file bugs, questions, feature requests and documentation gaps through one form. On-call engineers read every issue to decide whether it is theirs, which slows the response to real defects.

Approach

Logistic regression on title and body text, trained on the earlier 80% of 142,320 real GitHub issues (NLBSE’23) and tested on the later 20%, against labelling everything a bug and against title rules.

Result

Feature requests gain most, because users describe them in plain words. 88% of real bugs stay in the bug queue. Questions remain hardest: many read like bug reports.

The business problem

A software product with a public issue tracker gets bug reports, feature requests and support questions in the same place. Someone has to read each one, label it and send it to the owning team, and questions that reach engineering cost developer time. Maintainers who automate the first sort report first replies within minutes instead of days, while people keep the final decision.

I measured how accurately a labelling step sorts new issues into bug, feature, question or documentation, and how much of the support load it keeps out of engineering.

Team
Engineering on-call, a support or developer-relations team, product managers
Scale
Dozens to thousands of new issues a week
Systems
GitHub Issues, GitLab, Jira Service Management, Linear
Measure
Engineering hours on non-bug issues; time to first response on bugs

What was built

A labelling step that runs when an issue is opened. It reads the title and body, labels the issue as a bug, feature request, question or documentation gap, and assigns the owning team. Issues the model is unsure about keep a "needs triage" label for a person. Maintainers can change a label in one click, and changes are kept for retraining.

Issue openedTitle, body,code blocks removedLabelBug, feature,question or docsSure enough0.75+: label andteam set (71%)Needs triageBelow 0.75 (29%):a maintainer decides
Where the labelling step sits in the tracker. Nothing is closed or answered automatically.

Results

Bug58.7%16,708Feature request31.5%8,966Question5.6%1,603Documentation4.2%1,187
Bugs are 59% of new issues; questions and documentation together are 10%. Data: NLBSE'23 issue report classification, later 20% by issue id

The model labels 85.1% of issues correctly against 62.3% for title and template rules. The gain is largest on feature requests, which rules mostly miss because users describe what they want in plain words.

MethodAccuracyMacro F195% range (F1)
Label everything as a bug58.7%0.18
Title and template rules62.3%0.420.41 to 0.43
Text model85.1%0.740.73 to 0.74
Title and template rulesModelBug0.90Feature request0.84Question0.57Documentation0.63
Questions are the hardest type for both methods. Many read like bug reports written by someone unsure whether the behaviour is a bug.

Engineering time

Everything to engineeringno triage100%Title and template rules57%Model19%
With no triage every question and docs issue reaches engineering. Rules cut that to 57%; the model cuts it to 19%, while keeping 88% of real bugs in the bug queue.

Automatic labelling

40%60%80%100%0.400.500.600.700.800.90Accuracy of auto-labelled issuesShare auto-labelled0.75: 71% labelled, 93.3% rightConfidence bar
At a 0.75 bar, 71% of issues are labelled without a maintainer and 93.3% of those labels are right. 8,211 issues keep the needs-triage label.

Where it goes wrong

The confusion table shows where the labels go wrong. Most errors are questions labelled as bugs and the reverse: a user reporting unexpected behaviour is often asking whether it is intended.

Actual \ labelled asBugFeature requestQuestionDocumentation
Bug14,7411,267507193
Feature request7297,752241244
Question39121196734
Documentation13623857756
Under 200 chars7,111 issues82%200 to 1,00013,538 issues85%1,000 to 5,0006,733 issues88%Over 5,0001,079 issues90%
Short issues are harder: with under 200 characters of body the model has little more than the title to go on.
0.500.600.700.802,0005,00010,00030,00060,000113,856Macro F1
Accuracy rises quickly to about 30,000 past issues and then flattens, so a smaller project can start with a few thousand labelled issues and improve as it goes.

Data and method

Data
NLBSE'23 tool competition, issue report classification (Kallis, Izadi, Pascarella, Chaparro and Rani, 2023): real GitHub issues from public repositories, labelled bug, feature, question or documentation from their own labels
Split
The published test file of 142,320 issues, sorted by global GitHub issue id (which rises over time). The earlier 80% (113,856) trained the model; the later 20% (28,464) is the test period
Model
TF-IDF on word unigrams and bigrams of the title (counted twice) and the body with code blocks and links removed; logistic regression with balanced class weights
Baselines
Labelling everything as a bug, and rules on title wording and common issue-template phrases
Ranges
95% bootstrap ranges over test issues, 300 resamples

Words the model leans on

Issue typeWords the model leans on
Bugbug, not, reproduce, fix, crash, to reproduce, broken
Documentationdocumentation, docs, document, doc, readme, guide, documented
Feature requestfeature, add, implement, enhancement, allow, support, feat
Questionquestion, your question, is there, how, is it, hi, can

Limitations

What an engineering team would do next

Technical depth

The title is repeated in the input so its words carry more weight than a long body. Code blocks, HTML comments and links are stripped, and bodies are cut to 2,000 characters. Logistic regression with C = 4 and balanced class weights keeps the rarer question and documentation types from being swamped by bugs.

Analysis code (Python)

Financial impact

About 2 engineering hours a week returned at 300 new issues

Counted is engineering time spent reading questions and documentation issues that belong with support. Faster bug response is not priced.

InputValueSource
New issues a week300 commercial product with a public tracker
Questions and docs share of issues9.8%Measured
Reaching engineering: rules57%Measured
Reaching engineering: model19%Measured
Engineer minutes per misplaced issue10 read, comment, relabel
Engineer cost per hourA$110 loaded

Calculation

  1. Questions and docs issues a week29
  2. Fewer reaching engineering11 a week
  3. Engineering hours saved a week1.9
  4. Annual valueA$10,691

Annual value of engineering timeA$10,691

Open-source issue mixes differ from a commercial tracker; the bar should be set on the product’s own history.

Implementation and rollout

Six weeks from closed-issue export to labels applied above the bar

  1. Weeks 1 to 2

    Export

    A year of closed issues with their final labels from GitHub, GitLab, Jira or Linear.

    Engineering manager, Manaz

  2. Week 3

    Train and set the bar

    Train on the project’s own history; set the bar on the most recent quarter.

    Manaz

  3. Weeks 4 to 5

    Suggest-only

    Labels suggested as comments; maintainer changes measured.

    On-call engineers

  4. Week 6 onward

    Apply above the bar

    Labels and owning team set automatically; questions routed to support with a docs search link.

    Support and engineering leads

Tools

GitHub Issues, GitLab, Jira Service Management or Linear, GitHub Actions or a webhook running the model, Python and scikit-learn, Docs search link for questions

Team

  • Engineering manager owns the bar and labels
  • Support or developer-relations lead takes routed questions
  • Manaz model, action and monitoring

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
A real bug is routed to supportModel keeps 88% of bugs in the bug queue; held issues keep a needs-triage label.
Security reports land in a public trackerSecurity reports use a private route and never rely on a public-tracker label.
Labels drift as templates changeMonthly retraining on maintainer-corrected labels.
Contributors feel dismissed by automatic labelsOne-click label change and a short explanation comment.

Key takeaways

Key takeaways

  • Rules on titles and templates miss feature requests, which users describe in plain words.
  • The useful measure is engineering time on non-bugs, not raw accuracy.
  • Accuracy flattens after about 30,000 past issues, so a smaller project can start with a few thousand.
  • Questions are hardest for both methods, so they keep a human in the loop.