“Can I just use AI to compare these documents?”
Whether referring to general-purpose AI like ChatGPT and Copilot, legal-specific platforms like Harvey and Legora, or software with built-in comparison features like iManage and NetDocuments, this question is rapidly gaining traction across the legal sector.
And fair enough, too. Why wouldn’t you just use AI?
Put simply, accuracy in legal document comparison is non-negotiable. And when absolute accuracy is required, generative AI simply isn’t built for the task.
Here’swhy.
Document comparison is a zero-miss job
Lawyers compare documents to know, with 100% certainty, what changed before a client signs. It’s why firms run a fresh comparison even when opposing counsel provides tracked changes. Professional goodwill exists, but in legal practice, goodwill isn't a guarantee.
At its core, document comparison is a risk-management discipline. Its entire value rests on absolute completeness: every single character, line, and formatting shift caught without exception.
That is the non-negotiable bar any comparison tool must clear. Because generative AI is built on probabilistic prediction rather than deterministic tracking, it inherently carries a margin of error. In a zero-miss workflow, a margin of error is a liability legal practice simply cannot accept.
Probabilistic prediction vs. deterministic tracking
AI works by predicting the most likely answer rather than mechanically checking every character.
Think of it like tasking a junior lawyer to manually mark up two versions of a contract. In most cases they will catch almost everything, but because they are human, they can easily miss a single line. AI behaves the exact same way.
In most everyday tasks, getting things right "most of the time" is fine. In document comparison, missing a single word can redefine an entire obligation. That margin for error makes AI a structural liability for comparison.
Putting AI to the test in comparing documents
To see how generative AI handles a real comparison scenario, we put it to the test on a straightforward 18-page real-estate contract. We escalated the prompt each time to give ChatGPT every possible chance to succeed:
- Simple prompt: The AI found 11 “substantive” changes.
- Asking for a tracked-changes file: It found 75 changes.
- A dedicated comparison engine running on the exact same document found 143 changes.
- Detailed prompt naming every element to check: It found 95 changes.
Even after careful prompting, ChatGPT admitted it had missed changes, including moved sections and renumbered clauses.
Another issue encountered was that the output would be unworkable for a drafting lawyer. ChatGPT returned a side-by-side HTML file and plain text with tables stripped out, the table of contents deleted, and heading styles rewritten. It was nowhere near the clean Word or PDF redline a lawyer needs to work from.
Managing larger files for comparisons
That test was a best-case scenario: a short, clean, 18-page document. Real legal matters regularly run to hundreds or thousands of pages with thousands of complex edits.
AI gets noticeably less reliable as documents grow because of "context rot." The more text a model reads, the more its attention drifts. Add a scanned PDF into the mix, which lacks underlying document structure, and the AI has to manually reconstruct the layout before comparing it. This creates false positives, hallucinated edits, and unreadable redlines.
The larger and higher-stakes the document, the worse AI performs.
Newer AI models aren’t necessarily more reliable
We can’t say that the limitations are a temporary gap that the next release will fix because it is a fundamental design limitation.
We saw this exact pattern across multiple platforms. Copilot in Word couldn't generate a tracked-changes file at all. Claude was the strongest LLM we tested and, while higher fidelity, it was the slowest of the group and still couldn't render moved text properly and completely missed numbering and table-of-contents updates.
Tellingly, Claude even concluded its own analysis by stating that for a guaranteed, complete redline, a dedicated comparison tool is “the reference.” When the AI itself points you back to purpose-built software, it is telling you something structural about what it can and cannot do.
The right way to think about your tech stack and AI
None of this means AI has no place in legal work. Generative AI is genuinely remarkable at reading, summarising, drafting, and analysing concepts.
In fact, the strongest setup is using AI alongside a purpose-built comparison engine, so you get intelligence and speed without creating blind spots.
For drafting, summarising, and research, reach for AI. For catching every single change with zero exceptions, reach for the tool built to do it.
