The business problem
A finance or procurement team can only see where money goes if each purchase line sits in a spend category. In practice the category comes from the supplier master: one supplier, one category. That breaks for the suppliers that sell across categories and for every new supplier that arrives without a code, so the spend cube is cleaned by hand before each sourcing review. Organisations that classify on the line description, with a person reviewing the uncertain lines, report category accuracy in the 90s at the commodity level.
I measured how well a model that reads the description, not only the supplier, codes each purchase into a category, including suppliers it has never seen, against coding by supplier history.
- Team
- Accounts payable, procurement, management accountants
- Scale
- Hundreds to tens of thousands of bills and purchase orders a month
- Systems
- ERP purchasing data from SAP, Oracle or TechnologyOne, a spend analytics tool, UNSPSC or an internal category tree
- Measure
- Share of lines coded right first time; hours spent on reclassification
What was built
A coding step that reads the line description, the supplier and the buying team, and suggests a category. Above a confidence bar the code is applied; below it the top three suggestions are shown to a person, who picks one. Each choice is kept for retraining.
Results
On 15,297 later notices the text model picks the right segment 73.2% of the time, against 57.6% for the supplier default code that bookkeeping systems use. The right segment is in its top three 89.6% of the time.
| Level | Supplier default code | Text model | Top three | Value-weighted (model) |
|---|---|---|---|---|
| Segment (54) | 57.6% | 73.2% | 89.6% | 62.4% |
| Family (227) | 46.4% | 67.5% | 82.4% | 56.0% |
Automatic coding
Where it goes wrong
| UNSPSC segment | Notices | Text model | Supplier default |
|---|---|---|---|
| Management, business and administrative services | 4,673 | 85.6% | 81.7% |
| Engineering, research and technology services | 1,762 | 66.9% | 42.8% |
| IT, broadcasting and telecommunications | 1,586 | 76.2% | 60.7% |
| Vehicles and their accessories | 1,130 | 86.6% | 71.7% |
| Education and training services | 972 | 81.9% | 41.5% |
| Politics and civic affairs services | 697 | 59.5% | 30.7% |
| Building and facility construction and maintenance | 439 | 67.7% | 38.7% |
| Defence, law enforcement and safety equipment | 416 | 71.6% | 51.7% |
| Transport, storage and mail services | 283 | 70.3% | 49.5% |
| Furniture and furnishings | 272 | 90.8% | 81.6% |
Confident errors are often reasonable codes for an ambiguous description, where buyers themselves code similar work differently:
| Description (shortened) | Coded by the buyer | Model suggested | Confidence |
|---|---|---|---|
| Catalyst 2960-X 48 GigE PoE 370W + 2 x SFP+ | Structures and building components | IT, broadcasting and telecommunications | 0.92 |
| Purchase of vehicle batteries | Power generation and distribution machinery | Vehicles and their accessories | 0.88 |
| Computer Hardware | Office equipment and supplies | IT, broadcasting and telecommunications | 0.92 |
| Internal controls advisory services | Financial and insurance services | Management, business and administrative services | 0.92 |
| Labour Hire | Public utilities and public sector services | Management, business and administrative services | 0.93 |
| Hudson Infrastructure Analyst | Engineering, research and technology services | Management, business and administrative services | 0.80 |
Data and method
- Data
- AusTender contract notices published in FY2017-18 (Department of Finance, data.gov.au, CC BY 3.0 AU): 60,585 parent notices with a description, supplier, agency and the UNSPSC code chosen by the buyer, across 54 segments and 274 families
- Split
- By publish date. Notices before 2018-04-23 (45,288) trained the model; later notices (15,297) are the test period
- Model
- TF-IDF on word unigrams and bigrams of description, cleaned supplier name and agency. Logistic regression for segment; a linear support-vector model with softmax-scaled scores for family
- Baseline
- The supplier default: the code most often used for that supplier in the training period, else the commonest code overall
- Value-weighted
- Share of contract value in correctly coded notices
- Uncertainty
- With 15,297 test notices the 95% sampling range on each accuracy figure is about ±0.7 points
Limitations
- The labels are the codes buyers chose, which are themselves inconsistent. Some measured errors are cases where the model picked an equally defensible code.
- Government contract notices are larger and more formal than everyday bills. A business ledger has shorter descriptions and its own chart of accounts, so accuracy should be measured on its own history before automatic coding is switched on.
- Only one financial year is used, so seasonal changes in buying are not tested across years.
- UNSPSC segments stand in for a chart of accounts. Mapping to account codes adds a step that is usually one-to-one for most segments.
What a finance team would do next
- Export twelve months of coded bills with descriptions and suppliers
- Train on that history and set the bar on the latest quarter
- Run suggest-only for one month-end and count reclassifications
- Then: auto-code above the bar and send the rest to review with three suggestions
Technical depth
Supplier names are upper-cased and stripped of legal suffixes so the same supplier matches across notices. Sublinear term frequency stops long descriptions dominating. Classes with fewer than three training notices are dropped from training but kept in the test set, so they count as errors.