Manaz Afzal
Accounting & Finance · Bills, receipts and forms

Every purchase in the right spend category, including new suppliers

A coding step that reads the description, not only the supplier, tested on 15,297 later contract notices.

73.2%right segment; supplier default 58%
60%right for new suppliers; default 26%
64%coded automatically at 88% accuracy

Executive summary

Reading the description, not only the supplier, lifts coding accuracy from 58% to 73% and codes 64% of lines automatically at 88% accuracy.

73.2%of purchases coded to the right segment, against 57.6% for the supplier default
60%right for suppliers never seen before, against 26%
63 hof reclassification saved a month at 2,000 lines

Situation

Bookkeeping systems suggest the code a supplier had last time. Suppliers that sell many kinds of goods, and every new supplier, end up miscoded, and finance teams fix it at month-end.

Approach

A text model trained on 45,288 earlier government contract notices reads the description, supplier and buyer, and is tested on the 15,297 that followed, against the supplier-default rule.

Result

Segment accuracy rises from 57.6% to 73.2%. Above a 0.7 confidence bar 64% of lines are coded automatically at 88.1% accuracy; the rest go to a person with three suggestions.

The business problem

A finance or procurement team can only see where money goes if each purchase line sits in a spend category. In practice the category comes from the supplier master: one supplier, one category. That breaks for the suppliers that sell across categories and for every new supplier that arrives without a code, so the spend cube is cleaned by hand before each sourcing review. Organisations that classify on the line description, with a person reviewing the uncertain lines, report category accuracy in the 90s at the commodity level.

I measured how well a model that reads the description, not only the supplier, codes each purchase into a category, including suppliers it has never seen, against coding by supplier history.

Team
Accounts payable, procurement, management accountants
Scale
Hundreds to tens of thousands of bills and purchase orders a month
Systems
ERP purchasing data from SAP, Oracle or TechnologyOne, a spend analytics tool, UNSPSC or an internal category tree
Measure
Share of lines coded right first time; hours spent on reclassification

What was built

A coding step that reads the line description, the supplier and the buying team, and suggests a category. Above a confidence bar the code is applied; below it the top three suggestions are shown to a person, who picks one. Each choice is kept for retraining.

Bill or orderDescription,supplier, buyerSuggestCategory segmentand familySure enough0.7+: coded(64%)ReviewBelow 0.7: a personpicks from top three
Where the coding step sits between capture and the ledger.

Results

Management, business and administrative services31%4,673Engineering, research and technology services12%1,762IT, broadcasting and telecommunications10%1,586Vehicles and their accessories7%1,130Education and training services6%972Politics and civic affairs services5%697Building and facility construction and maintenance3%439Defence, law enforcement and safety equipment3%416
Spend is spread across many segments, so a single default code fits few suppliers. Data: AusTender contract notices, FY2017-18

On 15,297 later notices the text model picks the right segment 73.2% of the time, against 57.6% for the supplier default code that bookkeeping systems use. The right segment is in its top three 89.6% of the time.

LevelSupplier default codeText modelTop threeValue-weighted (model)
Segment (54)57.6%73.2%89.6%62.4%
Family (227)46.4%67.5%82.4%56.0%
Supplier default code58%Text model73%Text model, any of top three90%
Supplier default codeText modelSupplier seen before77%New supplier60%
For suppliers seen before, the default code is right 66% of the time. For the 21% of notices from new suppliers it has nothing to go on (26%); the model still reads the description (60%).
Supplier default code46%Text model67%Text model, any of top three82%
At the finer family level the gap is similar.

Automatic coding

40%60%80%100%0.30.40.50.60.70.80.9Accuracy of auto-coded noticesShare auto-coded0.7: 64% coded, 88.1% rightConfidence bar
At a 0.7 bar, 64% of notices are coded without review and 88.1% of those are right. For the 5,472 sent to a person, the right segment is in the top three 78% of the time.

Where it goes wrong

UNSPSC segmentNoticesText modelSupplier default
Management, business and administrative services4,67385.6%81.7%
Engineering, research and technology services1,76266.9%42.8%
IT, broadcasting and telecommunications1,58676.2%60.7%
Vehicles and their accessories1,13086.6%71.7%
Education and training services97281.9%41.5%
Politics and civic affairs services69759.5%30.7%
Building and facility construction and maintenance43967.7%38.7%
Defence, law enforcement and safety equipment41671.6%51.7%
Transport, storage and mail services28370.3%49.5%
Furniture and furnishings27290.8%81.6%
Supplier default codeText modelUnder $10k82%$10k to $100k73%$100k to $1m74%$1m to $10m68%Over $10m68%
The model leads at every contract size. Large contracts are harder for both methods because their descriptions cover several kinds of work.
1 to 2 words4,295 notices76%3 to 4 words5,512 notices76%5 to 8 words4,100 notices69%9 or more1,390 notices66%
Longer descriptions are harder, not easier: they usually describe mixed or bespoke work that spans categories.

Confident errors are often reasonable codes for an ambiguous description, where buyers themselves code similar work differently:

Description (shortened)Coded by the buyerModel suggestedConfidence
Catalyst 2960-X 48 GigE PoE 370W + 2 x SFP+Structures and building componentsIT, broadcasting and telecommunications0.92
Purchase of vehicle batteriesPower generation and distribution machineryVehicles and their accessories0.88
Computer HardwareOffice equipment and suppliesIT, broadcasting and telecommunications0.92
Internal controls advisory servicesFinancial and insurance servicesManagement, business and administrative services0.92
Labour HirePublic utilities and public sector servicesManagement, business and administrative services0.93
Hudson Infrastructure AnalystEngineering, research and technology servicesManagement, business and administrative services0.80

Data and method

Data
AusTender contract notices published in FY2017-18 (Department of Finance, data.gov.au, CC BY 3.0 AU): 60,585 parent notices with a description, supplier, agency and the UNSPSC code chosen by the buyer, across 54 segments and 274 families
Split
By publish date. Notices before 2018-04-23 (45,288) trained the model; later notices (15,297) are the test period
Model
TF-IDF on word unigrams and bigrams of description, cleaned supplier name and agency. Logistic regression for segment; a linear support-vector model with softmax-scaled scores for family
Baseline
The supplier default: the code most often used for that supplier in the training period, else the commonest code overall
Value-weighted
Share of contract value in correctly coded notices
Uncertainty
With 15,297 test notices the 95% sampling range on each accuracy figure is about ±0.7 points

Limitations

  • The labels are the codes buyers chose, which are themselves inconsistent. Some measured errors are cases where the model picked an equally defensible code.
  • Government contract notices are larger and more formal than everyday bills. A business ledger has shorter descriptions and its own chart of accounts, so accuracy should be measured on its own history before automatic coding is switched on.
  • Only one financial year is used, so seasonal changes in buying are not tested across years.
  • UNSPSC segments stand in for a chart of accounts. Mapping to account codes adds a step that is usually one-to-one for most segments.

What a finance team would do next

  • Export twelve months of coded bills with descriptions and suppliers
  • Train on that history and set the bar on the latest quarter
  • Run suggest-only for one month-end and count reclassifications
  • Then: auto-code above the bar and send the rest to review with three suggestions

Technical depth

Supplier names are upper-cased and stripped of legal suffixes so the same supplier matches across notices. Sublinear term frequency stops long descriptions dominating. Classes with fewer than three training notices are dropped from training but kept in the test set, so they count as errors.

Analysis code (Python)

Financial impact

About 63 hours of month-end rework saved per 2,000 lines

The saving comes from fewer miscoded lines reaching the ledger, offset by the time a person spends confirming the lines held for review.

InputValueSource
Coded lines per month2,000 mid-sized business or one entity of a group
Lines wrong with supplier default42.4%Measured
Share coded automatically at the bar64.2%Measured
Error rate of automatic codes11.9%Measured
Minutes to find and fix a miscoded line6 includes journal and approval
Minutes to confirm a held line0.5 pick from three suggestions
Loaded staff cost per hourA$55 accounts officer, on-costs included

Calculation

  1. Lines to fix today847
  2. Lines to fix with automatic coding153
  3. Lines held for a quick review715
  4. Net time saved a month63 hours
  5. Annual value at A$55 an hourA$41,903

Annual value of time savedA$41,903

Better category reporting, budget control and audit trails are real but are not counted here.

Implementation and rollout

Eight weeks from export to automatic coding, with a full month-end in shadow mode

  1. Weeks 1 to 2

    Export and baseline

    Twelve months of coded bills and purchase lines; measure the current supplier-default accuracy and reclassification rate.

    Finance systems lead, Manaz

  2. Weeks 3 to 4

    Train and backtest

    Train on the first nine months, test on the last three, set the confidence bar with the finance manager.

    Manaz, management accountant

  3. Weeks 5 to 7

    Shadow month-end

    Suggestions shown beside each line; the team codes as usual and every disagreement is logged.

    Accounts payable team

  4. Week 8 onward

    Automatic coding

    Lines above the bar are coded on entry; held lines show three suggestions; monthly retraining on confirmed codes.

    Accounts payable lead, Manaz on support

Tools

Xero or MYOB bill and purchase-order exports, Python (scikit-learn) model behind a small API, Make or n8n to write codes back through the accounting API, A review queue in the existing approval workflow

Team

  • Finance manager owns the bar and signs off go-live
  • Accounts payable lead reviews held lines and flags bad suggestions
  • Manaz data work, model, integration and monitoring

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Codes drift as the chart of accounts or suppliers changeMonthly retraining on confirmed codes; accuracy report per category with an alert when it falls below the bar.
A confident wrong code reaches the ledger unnoticedAutomatic codes are tagged; 50 automatic codes are checked each month, and high-value lines always go to review.
New categories with no historyLines from new categories fall below the bar and go to a person until enough examples exist.
Team over-trusts suggestions on held linesShow the confidence and the top three, not a single answer; track how often the first suggestion is overridden.

Key takeaways

Key takeaways

  • The supplier default is right only 58% of the time on this data and only 26% for new suppliers.
  • Reading the description lifts accuracy to 73% and to 90% when a person picks from three suggestions.
  • A confidence bar turns accuracy into a staffing decision: 64% of lines coded with no review at 88% accuracy.
  • The model should be retrained on each client’s own coded history before automatic coding is switched on.