SaaS & Tech · Inbox and ticket triage
Bugs to engineering, questions to support, before anyone reads the issue
A labelling step for a product's public issue tracker, tested on 28,464 later GitHub issues.
Executive summary
A labelling step on a public issue tracker cuts the share of questions and docs issues reaching engineering from 57% to 19%, with 85.1% of issues labelled correctly.
Situation
Customers file bugs, questions, feature requests and documentation gaps through one form. On-call engineers read every issue to decide whether it is theirs, which slows the response to real defects.
Approach
Logistic regression on title and body text, trained on the earlier 80% of 142,320 real GitHub issues (NLBSE’23) and tested on the later 20%, against labelling everything a bug and against title rules.
Result
Feature requests gain most, because users describe them in plain words. 88% of real bugs stay in the bug queue. Questions remain hardest: many read like bug reports.
The business problem
A software product with a public issue tracker gets bug reports, feature requests and support questions in the same place. Someone has to read each one, label it and send it to the owning team, and questions that reach engineering cost developer time. Maintainers who automate the first sort report first replies within minutes instead of days, while people keep the final decision.
I measured how accurately a labelling step sorts new issues into bug, feature, question or documentation, and how much of the support load it keeps out of engineering.
- Team
- Engineering on-call, a support or developer-relations team, product managers
- Scale
- Dozens to thousands of new issues a week
- Systems
- GitHub Issues, GitLab, Jira Service Management, Linear
- Measure
- Engineering hours on non-bug issues; time to first response on bugs
What was built
A labelling step that runs when an issue is opened. It reads the title and body, labels the issue as a bug, feature request, question or documentation gap, and assigns the owning team. Issues the model is unsure about keep a "needs triage" label for a person. Maintainers can change a label in one click, and changes are kept for retraining.
Results
The model labels 85.1% of issues correctly against 62.3% for title and template rules. The gain is largest on feature requests, which rules mostly miss because users describe what they want in plain words.
| Method | Accuracy | Macro F1 | 95% range (F1) |
|---|---|---|---|
| Label everything as a bug | 58.7% | 0.18 | |
| Title and template rules | 62.3% | 0.42 | 0.41 to 0.43 |
| Text model | 85.1% | 0.74 | 0.73 to 0.74 |
Engineering time
Automatic labelling
Where it goes wrong
The confusion table shows where the labels go wrong. Most errors are questions labelled as bugs and the reverse: a user reporting unexpected behaviour is often asking whether it is intended.
| Actual \ labelled as | Bug | Feature request | Question | Documentation |
|---|---|---|---|---|
| Bug | 14,741 | 1,267 | 507 | 193 |
| Feature request | 729 | 7,752 | 241 | 244 |
| Question | 391 | 211 | 967 | 34 |
| Documentation | 136 | 238 | 57 | 756 |
Data and method
- Data
- NLBSE'23 tool competition, issue report classification (Kallis, Izadi, Pascarella, Chaparro and Rani, 2023): real GitHub issues from public repositories, labelled bug, feature, question or documentation from their own labels
- Split
- The published test file of 142,320 issues, sorted by global GitHub issue id (which rises over time). The earlier 80% (113,856) trained the model; the later 20% (28,464) is the test period
- Model
- TF-IDF on word unigrams and bigrams of the title (counted twice) and the body with code blocks and links removed; logistic regression with balanced class weights
- Baselines
- Labelling everything as a bug, and rules on title wording and common issue-template phrases
- Ranges
- 95% bootstrap ranges over test issues, 300 resamples
Words the model leans on
| Issue type | Words the model leans on |
|---|---|
| Bug | bug, not, reproduce, fix, crash, to reproduce, broken |
| Documentation | documentation, docs, document, doc, readme, guide, documented |
| Feature request | feature, add, implement, enhancement, allow, support, feat |
| Question | question, your question, is there, how, is it, hi, can |
Limitations
- Labels come from each project's own tags, which are applied unevenly. Some "bugs" are questions in disguise, so measured accuracy understates how often a maintainer would agree with the model.
- The data covers open-source projects. A commercial product tracker has a different mix of customers, issue templates and products, so the bar should be set on its own history.
- Repository names are not in the file, so the split is by time across all projects. Accuracy on a single new project with no history will start lower.
- Only the four types are covered. Security reports need their own private route and should never rely on a public-tracker label.
What an engineering team would do next
- Export a year of closed issues with their final labels
- Train on the project's own history and set the bar on the most recent quarter
- Run in suggest-only mode for a month and measure label changes by maintainers
- Then: apply labels above the bar, route questions to support with a link to the docs search
Technical depth
The title is repeated in the input so its words carry more weight than a long body. Code blocks, HTML comments and links are stripped, and bodies are cut to 2,000 characters. Logistic regression with C = 4 and balanced class weights keeps the rarer question and documentation types from being swamped by bugs.