Manaz Afzal

Legal & Professional Services · Contract portfolio tagging

Every clause in the portfolio, tagged by what it says

A tagging step that reads each provision on its own text and assigns one of a hundred clause types, tested on 10,000 provisions from later filings.

87.7%tagged with the right clause type; name search 23%
78%tagged automatically at 95.8% accuracy
96%right type within the top three

Executive summary

Reading each provision tags 88% of clauses with the right type across a hundred clause types, against 23% for clause-name search, and writes 78% of tags with no human step at 96% accuracy.

87.7%of provisions tagged with the right clause type, against 22.5% for name search
78%tagged automatically at 95.8% accuracy
88 hof tagging time saved a month at 300 new agreements

Situation

Signed agreements and precedents sit in the document system untagged or tagged by heading, so portfolio questions such as every assignment needing consent mean opening files one by one.

Approach

A text model trained on 60,000 provisions from earlier filings across a hundred clause types, tested on 10,000 later provisions against clause-name search.

Result

Accuracy rises from 22.5% to 87.7%. Above a 0.8 bar 78% of provisions are tagged automatically; the rest reach a reviewer with three suggestions that contain the right type 87% of the time.

The problem

A company with a large contract portfolio needs to know what each clause says: which agreements cap liability, which renew automatically, which allow assignment. Contract repositories search on clause headings, which vary from one drafter to the next, so portfolio reviews fall back on reading.

I measured how accurately a tagging step that reads each provision on its own text assigns it one of a hundred clause types, against searching by clause name.

Team
Knowledge lawyers, contract managers, in-house counsel
Systems
iManage or NetDocuments, SharePoint in Microsoft 365, and contract lifecycle tools such as Ironclad or Icertis
Measure
Share of provisions tagged with the right clause type; reviewer time per agreement

What I built

A tagging step that reads each provision on its own text, without relying on its heading, and assigns one of a hundred clause types with a confidence score. Confident tags are written straight to the document system; the rest go to a reviewer with the three likeliest types.

AgreementSplit intoprovisionsReadClause type andconfidenceSure enough0.8+: tag written(78%)ReviewerBelow the bar: pickfrom top three
Tags feed the clause library, review checklists and portfolio searches such as every assignment clause that needs consent.

This complements the clause finder study, which locates specific risk clauses inside one contract. Tagging works across the whole portfolio.

Results

On 10,000 provisions from later filings, the text model tags 87.7% correctly (95% range 87.2% to 88.3%); the right type is within its top three 96.4% of the time. Searching each provision for the clause name, the usual first attempt, gets 22.5% right and finds nothing at all in 5.7% of provisions.

Most common clause type only6%Clause name search23%Text model88%Text model, within top three96%

Automatic tagging

50%60%70%80%90%100%0.30.40.50.60.70.80.9Accuracy of auto-taggedShare auto-tagged0.8: 78% tagged,95.8% rightConfidence bar
At a 0.8 bar, 78% of provisions are tagged with no human step and 95.8% of those tags are right. For the 2,243 held for review, the right type is in the top three 87% of the time.

The clauses reviewers check first

Name searchText modelConfidentiality100%Non-Disparagement100%Severability99%Survival97%Intellectual Property97%Waiver Of Jury Trials96%Indemnifications96%Governing Laws96%Arbitration96%Change In Control93%Assignments89%Terminations81%
Governing law, confidentiality, indemnities, jury-trial waivers and severability are found more than 95% of the time. Change-of-control and IP clauses are where name search does best, because their wording names them.
Clause typeTest provisionsFound (recall)Tags that were right (precision)Name search recall
Governing Laws58296.0%90.0%0.9%
Indemnifications15296.1%86.4%13.8%
Terminations11881.4%83.5%8.5%
Confidentiality167100.0%88.4%7.2%
Assignments19588.7%83.2%15.9%
Change In Control8793.1%87.1%63.2%
Waiver Of Jury Trials11196.4%96.4%0.9%
Arbitration4795.7%80.4%4.3%
Intellectual Property6496.9%91.2%32.8%
Non-Disparagement46100.0%93.9%4.3%
Survival22297.3%90.8%0.0%
Severability41298.5%94.4%0.0%

Where it goes wrong

Most errors fall between clause types that lawyers themselves use interchangeably: applicable law and governing law, waivers and no waivers, amendments and modifications. A firm’s own taxonomy usually merges these, which lifts accuracy further.

Clause typeMost often tagged asProvisions
Applicable LawsGoverning Laws29
No WaiversWaivers23
Tax WithholdingsWithholdings18
ModificationsAmendments18
DefinitionsDefined Terms17
IntegrationEntire Agreements16
SuccessorsBinding Effects14
WarrantiesSurvival14
Hardest clause typesTest provisionsF1
Applicable Laws530.35
Jurisdictions290.40
Venues200.50
Miscellaneous790.55
Agreements480.59
Positions320.60
Governing Laws582 provisions5.8%Counterparts490 provisions4.9%Notices422 provisions4.2%Severability412 provisions4.1%Entire Agreements376 provisions3.8%Amendments224 provisions2.2%Survival222 provisions2.2%Headings220 provisions2.2%Compliance With Laws208 provisions2.1%General203 provisions2.0%Assignments195 provisions1.9%Taxes185 provisions1.8%
Boilerplate dominates. Governing law, counterparts and notices together make up about 15% of provisions, which is why the macro F1 (0.83) sits below accuracy.

Data and method

Data
LEDGAR from LexGLUE (Tuggener et al., 2020; Chalkidis et al., 2022; CC BY-SA 4.0): provisions from material contracts filed with the US SEC on EDGAR, each labelled with one of 100 clause types derived from its own heading, with the heading removed from the text
Split
LexGLUE’s chronological split: 60,000 provisions train, 10,000 set the regularisation, 10,000 later provisions test
Model
TF-IDF on word unigrams and bigrams with sublinear term frequency; logistic regression, C = 20 chosen on validation
Baseline
Clause-name search: the clause type whose name appears earliest in the first 400 characters of the provision
Provision length
median 78 words
Uncertainty
Bootstrap 95% range on accuracy over test provisions

Limitations

  • The contracts are US public-company filings. Australian agreements use some different clause conventions, such as PPSA and GST provisions, which need labelled examples from the firm’s own documents.
  • Each provision carries one label. Long provisions that combine, for example, termination and survival are tagged with one type and the second is shown in the top three.
  • Tagging says what a clause is, not whether it is acceptable. Judging a clause against the firm’s playbook is a separate review step.
  • Provisions must be split correctly first; scanned agreements need OCR and a splitting step before tagging.

How I would take it live

  • Take a set of signed agreements already tagged by the knowledge team, or tag a set with the firm’s own clause list
  • Retrain on the firm’s taxonomy and agree the bar with the knowledge lawyer
  • Tag the precedent bank first, where errors are cheap and reviewed
  • Then tag new agreements on filing and add portfolio searches and alerts

Financial impact

About 88 hours of clause tagging saved a month at 300 new agreements

The saving is the knowledge team’s tagging time. Confident tags are written directly; uncertain ones become a quick pick from three.

InputValueSource
New agreements a month300 commercial firm or corporate legal team
Provisions per agreement40
Minutes to tag a provision by hand0.5
Share tagged automatically77.6%Measured
Accuracy of automatic tags95.8%Measured
Minutes to pick from three suggestions0.2
Loaded cost per hourA$110 knowledge lawyer or senior paralegal

Calculation

  1. Tagging time today100 hours
  2. Time with the model (picks and corrections)12.3 hours
  3. Net time saved a month88 hours
  4. Annual valueA$115,827

Annual value of time savedA$115,827

Tagging the existing precedent bank is a one-off gain on top, and portfolio searches that become possible are not counted here.

Implementation and rollout

Eight weeks from document export to tagged portfolio

  1. Weeks 1 to 2

    Taxonomy and labelled set

    Agree the firm’s clause list with the knowledge lawyer; export agreements already tagged, or tag a set of them by hand.

    Knowledge lawyer, Manaz

  2. Weeks 3 to 4

    Split and train

    Split agreements into provisions, train on the firm’s taxonomy, test on the latest agreements and set the bar.

    Manaz

  3. Weeks 5 to 6

    Precedent bank

    Tag the precedent bank with every tag reviewed; disagreements logged and used to retrain.

    Knowledge team

  4. Week 7 onward

    Live tagging

    New agreements tagged on filing; reviewer queue for those below the bar; monthly accuracy report.

    Knowledge lawyer, Manaz on support

Tools

SharePoint or OneDrive in Microsoft 365, or LEAP, Actionstep or Smokeball documents, Provision splitter and OCR for scanned agreements, Python (scikit-learn) model inside the firm’s tenancy, Tags written back as document metadata for search and alerts

Team

  • Knowledge lawyer owns the taxonomy and the bar
  • Knowledge team or paralegals review held provisions
  • Manaz data work, model, integration and monitoring

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Client confidentialityThe model runs inside the firm’s own environment; no agreement text leaves it.
A wrong tag hides a clause from searchSearches can include provisions where the type is in the top three; key clause types are reviewed below a higher bar.
Australian clause conventionsPPSA, GST and state-specific provisions are added to the taxonomy with labelled examples from the firm’s agreements.
Taxonomy driftNew clause types are added with examples and accuracy is reported by type each month.

Key takeaways

Key takeaways

  • Clause-name search tags 23% of provisions correctly; reading the provision lifts that to 88%.
  • A confidence bar writes 78% of tags automatically with 96% accuracy.
  • Most errors are between clause types lawyers treat as the same, which a firm’s own taxonomy merges.
  • Accuracy is measured on the firm’s own agreements and clause list before tags are written automatically.