Manaz Afzal

Legal · Document automation

A review assistant that fills the contract checklist

Every checklist item comes back with the three passages most likely to answer it, so the reviewer reads about 6% of the contract.

94.3%right passage in the top three; keyword search 52.6%
81%ranked first
6%of the text read per item

Executive summary

For 94% of checklist items the right passage is in the assistant’s top three, so a reviewer reads about 6% of a contract per item instead of searching all of it.

94.3%right passage in the top three on 102 held-out contracts (95% CI 92.6% to 95.8%)
52.6%for keyword search on the clause’s own name
81%of items with the right passage ranked first

Situation

Commercial teams review supplier and licence agreements against the same checklist. Judging a clause takes expertise; finding it takes reading, and keyword search misses half the time because contracts rarely use the checklist’s words.

Approach

Contracts split into passages; one text model per checklist item, trained on 408 lawyer-labelled CUAD contracts and scored on the 102 in the official test split.

Result

The assistant leads keyword search by 42 points (95% CI 38 to 45). It is weakest on rare clauses phrased many ways, such as liquidated damages (71%). 71 of 1,244 pairs still need finding by reading.

The business problem

A legal team reviewing commercial contracts works through a checklist: change of control, assignment, termination, liability caps. Firms using review tools such as Kira or Luminance report first-pass review times cut by a third to a half, because the tool finds the passages and the lawyer reads them.

I measured how often a review assistant puts the right passage for each checklist item in its top three, against keyword search, so the reviewer reads a small share of each contract.

Team
In-house counsel or a contracts manager, often one person
Scale
Tens to a few hundred agreements a month, many on the other side’s paper
Systems
SharePoint or Google Drive, Word, a checklist in Excel; sometimes Ironclad or LexisNexis
Constraint
Confidential documents stay in the firm’s tenancy; a lawyer signs off every item

What was built

The contract is split into passages. A text model trained on 408 lawyer-labelled contracts scores every passage for every checklist item. The review sheet lists the top three passages per item with a link back to the page; items with no likely passage are marked “not found” for the reviewer to confirm. The reviewer reads about 6% of the text per item instead of all of it.

UploadPDF or Word fromSharePoint or DriveSplitPassages withpage referencesScore41 checklist itemsper passageReview sheetTop three passagesper itemLawyerConfirms, edits,signs off
Review workflow. The assistant finds; the reviewer decides.

Results

Read the first three passages25%Keyword search, top three53%Assistant, first passage81%Assistant, top three94%
Scored on the 102 contracts in CUAD’s official test split, which the model never saw: 94.3% of 1,244 item and contract pairs (95% CI 92.6% to 95.8%). Data: CUAD v1, Hendrycks et al. (2021), CC BY 4.0
Governing LawAnti-AssignmentTermination For ConvenienceCap On LiabilityInsuranceAudit RightsNon-CompeteExclusivityPassage 1Passage 50Darker: higher model score. Outlined: the passage lawyers labelled for that item.
One test contract, a co-branding agreement, as the model sees it. Each column is a passage; outlines mark the lawyers’ labels.
Keyword searchAssistantGoverning Law100%Anti-Assignment100%Cap On Liability100%Termination For Convenience100%Change Of Control96%Non-Compete96%Exclusivity100%Audit Rights100%Insurance97%License Grant90%
Common checklist items: the assistant’s lead over keyword search is widest where clauses are phrased in many ways.

Where it struggles

Weakest items (10+ test contracts)Test contractsAssistant top 3Keyword top 3
Liquidated Damages1471.4%50.0%
Competitive Restriction Exception1675.0%43.8%
Volume Restriction1782.4%0.0%
Ip Ownership Assignment2382.6%52.2%
Post-Termination Services2982.8%37.9%
Document Name10284.3%23.5%

71 of 1,244 pairs still need finding by reading. Rare clauses phrased in many ways miss most often.

Data and method

Source
CUAD v1 (Hendrycks, Burns, Chen and Ball, 2021): commercial contracts filed with the US SEC, labelled by lawyers
Size
510 contracts, 41 clause types, 13,823 labelled spans, 53,001 passages
Length
Median 33,052 characters in 66 passages; longest 337,344
Split
CUAD official: 408 train, 102 test

Baselines: reading the first three passages (25%) and keyword search on each item’s name (53%). The assistant’s gain over keyword search is 42% points (95% CI 38% to 45%).

0.500.550.600.650.700.752550100200408Mean PR-AUC
More labelled contracts still help, slowly.
Checklist itemWords the model leans on
Governing Lawgoverned, laws of, laws, the laws, governed by
Non-Competecompetitive, competitor, competing, competitors, compete
Termination For Convenienceterminate, termination, notice, written notice, any time
Cap On Liabilityliability, damages, consequential, remedy, exceed
Change Of Controlof control, change of, change, control, merger
Anti-Assignmentassignment, assign, transfer, void, exhibit 16

Assumptions and limitations

  • US public-company contracts; a firm’s own templates need a check on a few hundred of its own documents first
  • The assistant finds passages; it does not judge whether terms are acceptable
  • Scanned PDFs need OCR first, and OCR errors lower recall
  • Rare items such as most-favoured-nation clauses have few training examples

What a team would do next

  • Week 1: pick 200 to 300 past agreements and the checklist the team already uses
  • Weeks 2 to 3: label or import existing review notes; refit and test on held-out agreements
  • Weeks 4 to 8: reviewers use the sheet on live reviews and log misses
  • Then: add items or playbook positions where reviewers ask for them

Technical depth

Passages are scored per item by logistic regression on TF-IDF word and two-word features, one model per item; results use the official split with no tuning on test contracts.

Analysis code (Python)

Financial impact

About 34 hours of review time a month at 60 contracts

The saving is time spent locating clauses. Judging whether a term is acceptable stays with the reviewer and is not counted.

InputValueSource
Contracts reviewed a month60 in-house team
Checklist items found per contract12.2Measured 1,244 pairs over 102 test contracts
Minutes to locate an item by reading4 median contract about 12 pages
Minutes to confirm a suggested passage1
Right passage in top three94.3%Measured
Reviewer cost per hourA$150 in-house counsel, loaded

Calculation

  1. Minutes saved per item2.77
  2. Minutes saved per contract34
  3. Hours saved a month34
  4. Annual valueA$60,854

Annual value of review time savedA$60,854

Faster turnaround on the other side’s paper and fewer missed clauses are not priced.

Implementation and rollout

Eight weeks from past agreements to live reviews on the firm’s own paper

  1. Week 1

    Collect

    Pick 200 to 300 past agreements and the checklist the team already uses.

    Contracts manager, Manaz

  2. Weeks 2 to 3

    Label and refit

    Import existing review notes as labels; refit and test on held-out agreements.

    Manaz, a reviewer for spot checks

  3. Weeks 4 to 8

    Live reviews

    Reviewers use the sheet on live work and log misses; weekly top-three accuracy report.

    Legal team

  4. Then

    Extend

    Add items and playbook positions where reviewers ask for them.

    Head of legal

Tools

SharePoint or Google Drive, Word and the existing Excel checklist, Python (scikit-learn) inside the firm’s tenancy, OCR for scanned PDFs

Team

  • Head of legal or contracts manager owns the checklist and sign-off
  • Reviewers confirm passages and log misses
  • Manaz model, review sheet and integration

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Confidential contracts leave the firmModel runs inside the firm’s own tenancy; no external API.
A missing clause is read as absent“Not found” is a prompt to confirm, and the reviewer signs off every item.
Scanned PDFs lower recallOCR quality check before scoring; low-quality scans flagged for full reading.
US public-company contracts differ from the firm’s templatesRefit and test on a few hundred of the firm’s own agreements first.

Key takeaways

Key takeaways

  • Finding a clause and judging it are different jobs; automate the first, keep a lawyer on the second.
  • Keyword search finds the right passage only about half the time; a trained ranker reaches 94% in the top three.
  • Rare clauses phrased many ways are where misses concentrate, so they deserve extra labelled examples.
  • A reviewer reads about 6% of the text per item, which is where the time saving comes from.