The Summary Looked Perfect. The Deal Fell Apart.
A product manager shares a lengthy vendor contract with her team and asks the AI to summarize it. The summary comes back clean: payment terms, delivery schedule, key milestones. Everyone signs off. Six months later, an indemnification clause buried in section 14 costs the company a lot of money. The clause was in the document. The AI read every word. It just decided those words weren’t important enough to surface.
This is not a hallucination problem. The AI didn’t invent anything. It made a judgment call about salience, and it made that call based on criteria that have nothing to do with your specific situation, your risk tolerance, or what you actually needed to know.
Understanding why this happens, and what to do about it, is more useful than simply distrusting AI summarizers. You’re probably going to keep using them. You should. They’re genuinely helpful for a lot of tasks. But you need to understand what they’re optimizing for.
What “Important” Means to a Language Model
When a large language model summarizes a document, it isn’t identifying importance the way you would. It’s doing something closer to identifying centrality. Words, sentences, and ideas that appear frequently, that connect to many other parts of the document, and that match patterns the model has seen summarized before in its training data, those get surfaced. Words that appear once, in unfamiliar legal phrasing, near the end of a long document, in a clause that looks structurally minor, those get dropped.
The problem is that important information is often structurally peripheral. Exceptions are important. Carve-outs are important. The one sentence that contradicts everything else in the document is, by definition, statistically unusual. That’s exactly the kind of content that centrality-based summarization deprioritizes.
This connects to a broader issue worth understanding: the prompt you write isn’t the prompt the model reads. The model applies its own interpretive layer before it processes your instructions. Even if you ask it to “highlight unusual clauses” or “flag anything concerning,” the model is filtering through its own sense of what counts as unusual or concerning, which was shaped by training data, not by your context.
The Categories of Content Summarizers Consistently Miss
Once you know what to look for, the pattern becomes predictable. AI summarizers tend to do well with the body of a document and poorly with its edges.
Exceptions and negations. “The warranty does not cover” is one of the most important phrases in a product agreement. It’s also easy to compress away because the surrounding context is all about the warranty, and the summary captures that the document discusses warranty terms. The negation disappears.
Conditionals that depend on rare events. If a contract clause only triggers under specific circumstances, the model may rate it as low-relevance because the trigger condition seems unlikely. You might disagree, especially if that trigger condition is exactly the risk scenario you’re trying to assess.
Information in footnotes, appendices, and cross-references. Models processing long documents often weight later content less. Defined terms that appear in an appendix but govern the entire agreement can vanish from summaries entirely.
Anything that requires domain expertise to recognize as significant. A compliance officer would immediately flag a particular regulatory citation. A generalist model might treat it as background context. The model doesn’t know what you know, and it doesn’t know what it doesn’t know.
Deliberate ambiguity. Some language in contracts is vague on purpose. A good lawyer spots it; a summarizer often smooths it over, converting “reasonable efforts” (a term with real legal weight and real ambiguity) into something that reads as settled and clear.
How to Use AI Summarizers Without Getting Burned
The answer isn’t to stop using summarizers. It’s to stop using them as a replacement for reading and start using them as a complement to it.
The most useful reframe: treat the AI summary as a map, not the territory. It gives you a rough orientation so you can read the document faster and more strategically. It is not a substitute for reading.
A few practices that actually help:
Use summarizers for orientation, then read the full document for anything with real stakes. If the document is long and low-risk, a summary may be enough. If there are legal, financial, or safety implications, the summary is just your starting point.
Ask the model to do targeted extraction instead of open-ended summarization. “List every clause that creates an obligation on our part” or “find every instance of ‘except,’ ‘provided that,’ or ‘notwithstanding’” tends to surface edge content more reliably than “summarize this document.”
Ask what’s missing, not just what’s there. “What topics does this document cover that you didn’t mention in the summary?” is a useful follow-up prompt. It’s imperfect, but it sometimes surfaces what got dropped.
Build checklists based on document type. If you regularly review vendor contracts, you already know the five things that matter most: liability caps, IP ownership, termination rights, data handling, and indemnification. Give the model that checklist explicitly. “Does this contract address each of the following? For each one, quote the relevant language.” Extraction with a specific target beats open-ended summarization almost every time.
The Confidence Gap Is the Real Problem
Here’s the thing that makes this genuinely dangerous: AI summaries look authoritative. They’re well-written, coherent, and confident. There’s no signal that something was dropped. The summary doesn’t say “I omitted section 14 because it seemed peripheral.” It just presents what it chose to surface as if it were the whole picture.
Hallucinations, for all their problems, at least have the quality of being wrong in ways that occasionally get caught. A summary that is technically accurate but structurally incomplete is much harder to audit. You don’t know what you don’t know.
The right mental model for AI summarization in 2025 is something like a very smart, very fast junior analyst who has read a lot of documents but doesn’t know your business, doesn’t know your risk profile, and genuinely cannot tell you what they decided wasn’t worth mentioning. They’re useful. You’d be glad to have them. But you wouldn’t send their first draft to the board without reading it yourself.
Use AI to read faster. Don’t use it to read less.
