Legal & Professional Services · Contract portfolio tagging
Every clause in the portfolio, tagged by what it says
A tagging step that reads each provision on its own text and assigns one of a hundred clause types, tested on 10,000 provisions from later filings.
Executive summary
Reading each provision tags 88% of clauses with the right type across a hundred clause types, against 23% for clause-name search, and writes 78% of tags with no human step at 96% accuracy.
Situation
Signed agreements and precedents sit in the document system untagged or tagged by heading, so portfolio questions such as every assignment needing consent mean opening files one by one.
Approach
A text model trained on 60,000 provisions from earlier filings across a hundred clause types, tested on 10,000 later provisions against clause-name search.
Result
Accuracy rises from 22.5% to 87.7%. Above a 0.8 bar 78% of provisions are tagged automatically; the rest reach a reviewer with three suggestions that contain the right type 87% of the time.
The problem
A company with a large contract portfolio needs to know what each clause says: which agreements cap liability, which renew automatically, which allow assignment. Contract repositories search on clause headings, which vary from one drafter to the next, so portfolio reviews fall back on reading.
I measured how accurately a tagging step that reads each provision on its own text assigns it one of a hundred clause types, against searching by clause name.
- Team
- Knowledge lawyers, contract managers, in-house counsel
- Systems
- iManage or NetDocuments, SharePoint in Microsoft 365, and contract lifecycle tools such as Ironclad or Icertis
- Measure
- Share of provisions tagged with the right clause type; reviewer time per agreement
What I built
A tagging step that reads each provision on its own text, without relying on its heading, and assigns one of a hundred clause types with a confidence score. Confident tags are written straight to the document system; the rest go to a reviewer with the three likeliest types.
This complements the clause finder study, which locates specific risk clauses inside one contract. Tagging works across the whole portfolio.
Results
On 10,000 provisions from later filings, the text model tags 87.7% correctly (95% range 87.2% to 88.3%); the right type is within its top three 96.4% of the time. Searching each provision for the clause name, the usual first attempt, gets 22.5% right and finds nothing at all in 5.7% of provisions.
Automatic tagging
The clauses reviewers check first
| Clause type | Test provisions | Found (recall) | Tags that were right (precision) | Name search recall |
|---|---|---|---|---|
| Governing Laws | 582 | 96.0% | 90.0% | 0.9% |
| Indemnifications | 152 | 96.1% | 86.4% | 13.8% |
| Terminations | 118 | 81.4% | 83.5% | 8.5% |
| Confidentiality | 167 | 100.0% | 88.4% | 7.2% |
| Assignments | 195 | 88.7% | 83.2% | 15.9% |
| Change In Control | 87 | 93.1% | 87.1% | 63.2% |
| Waiver Of Jury Trials | 111 | 96.4% | 96.4% | 0.9% |
| Arbitration | 47 | 95.7% | 80.4% | 4.3% |
| Intellectual Property | 64 | 96.9% | 91.2% | 32.8% |
| Non-Disparagement | 46 | 100.0% | 93.9% | 4.3% |
| Survival | 222 | 97.3% | 90.8% | 0.0% |
| Severability | 412 | 98.5% | 94.4% | 0.0% |
Where it goes wrong
Most errors fall between clause types that lawyers themselves use interchangeably: applicable law and governing law, waivers and no waivers, amendments and modifications. A firm’s own taxonomy usually merges these, which lifts accuracy further.
| Clause type | Most often tagged as | Provisions |
|---|---|---|
| Applicable Laws | Governing Laws | 29 |
| No Waivers | Waivers | 23 |
| Tax Withholdings | Withholdings | 18 |
| Modifications | Amendments | 18 |
| Definitions | Defined Terms | 17 |
| Integration | Entire Agreements | 16 |
| Successors | Binding Effects | 14 |
| Warranties | Survival | 14 |
| Hardest clause types | Test provisions | F1 |
|---|---|---|
| Applicable Laws | 53 | 0.35 |
| Jurisdictions | 29 | 0.40 |
| Venues | 20 | 0.50 |
| Miscellaneous | 79 | 0.55 |
| Agreements | 48 | 0.59 |
| Positions | 32 | 0.60 |
Data and method
- Data
- LEDGAR from LexGLUE (Tuggener et al., 2020; Chalkidis et al., 2022; CC BY-SA 4.0): provisions from material contracts filed with the US SEC on EDGAR, each labelled with one of 100 clause types derived from its own heading, with the heading removed from the text
- Split
- LexGLUE’s chronological split: 60,000 provisions train, 10,000 set the regularisation, 10,000 later provisions test
- Model
- TF-IDF on word unigrams and bigrams with sublinear term frequency; logistic regression, C = 20 chosen on validation
- Baseline
- Clause-name search: the clause type whose name appears earliest in the first 400 characters of the provision
- Provision length
- median 78 words
- Uncertainty
- Bootstrap 95% range on accuracy over test provisions
Limitations
- The contracts are US public-company filings. Australian agreements use some different clause conventions, such as PPSA and GST provisions, which need labelled examples from the firm’s own documents.
- Each provision carries one label. Long provisions that combine, for example, termination and survival are tagged with one type and the second is shown in the top three.
- Tagging says what a clause is, not whether it is acceptable. Judging a clause against the firm’s playbook is a separate review step.
- Provisions must be split correctly first; scanned agreements need OCR and a splitting step before tagging.
How I would take it live
- Take a set of signed agreements already tagged by the knowledge team, or tag a set with the firm’s own clause list
- Retrain on the firm’s taxonomy and agree the bar with the knowledge lawyer
- Tag the precedent bank first, where errors are cheap and reviewed
- Then tag new agreements on filing and add portfolio searches and alerts