AI summarization: preserve the numbers, exceptions, and uncertainty

A summary can be beautifully written and still change the meaning of its source. It may turn a tentative proposal into a decision, merge two unrelated figures, or omit the exception that determines whether a rule applies. These mistakes are especially difficult to notice because the shorter text often sounds clearer and more confident than the original.

Reliable AI summarization begins by deciding what must survive compression. The goal is not merely fewer words. It is a shorter representation that supports the reader's task without introducing claims, certainty, or relationships that the source does not establish. That requires a review process focused on meaning as well as readability.

Distinguish source faithfulness from general plausibility

Research by Maynez and colleagues examines unfaithful content in abstractive summaries and the limits of common evaluation measures. A central practical lesson is that fluent wording and surface similarity are insufficient evidence that a summary accurately reflects its input. The paper studies particular systems; the workflow below is original application guidance.

Read the original research paper on arXiv

A statement can be generally plausible and still be unsupported by the document being summarized. If a meeting note says a team discussed a migration, a summary should not say the migration was approved merely because approval would make sense. Source faithfulness requires preserving the actual status of the claim.

The reverse problem also matters. A source can contain an error. A faithful summary may reproduce the source's claim while making clear who asserted it, rather than silently presenting it as independently verified truth. Decide whether the task is summarization, fact checking, or both, and keep those responsibilities distinguishable.

Write a summary contract for the intended reader

Consider a hypothetical engineering team preparing a weekly project briefing. The source material includes meeting notes, issue updates, and a test report. The reader needs decisions made, unresolved blockers, assigned actions, and changes to the delivery plan. They do not need a chronological retelling of every discussion.

Define required fields and acceptable omissions. Each action should preserve its owner and status when those are stated. Unassigned work should remain unassigned. A proposed date should not become a committed deadline. The contract should explicitly allow the model to report that a required detail is absent.

Choose a length budget that fits the task. If the input contains many consequential exceptions, an extremely short limit forces a choice about what to omit. Make that choice deliberately. A concise overview with links to a detailed exception list may be more useful than a single paragraph that hides all complexity.

Build an evidence ledger before polishing prose

For a demanding workflow, first extract candidate claims with source locations. Record the entity, action or observation, status, quantity, and relevant qualification. This intermediate ledger makes it easier to inspect what the model believes the source says before asking it to compress those claims into readable prose.

The ledger is not automatically correct. Review samples against the original documents and preserve the distinction between quoted source spans and generated interpretations. A source identifier should lead to the actual passage, not merely to a document that happens to discuss the same subject.

Then compose the summary from the reviewed evidence. This separation can help diagnose errors: did extraction miss the exception, or did composition drop it? Without an intermediate record, teams may repeatedly adjust the final prompt while never identifying where meaning was lost.

Numbers need their surrounding nouns

In an illustrative test report, 18 of 20 requests complete under a target latency, while a separate sample contains 200 total requests. A careless summary might combine the numerator from one statement with the denominator from another. The individual numbers are copied correctly, yet the resulting claim is false.

Review quantities together with their units, populations, periods, and comparison baselines. A percentage increase differs from a percentage-point increase. An average over successful requests differs from an average over all requests. A result from a staging environment should not quietly become a production measurement.

For derived calculations, use a deterministic calculation step and record the inputs. The summary should distinguish a figure stated by the source from a figure calculated during preparation. If the source values are ambiguous or inconsistent, surface the ambiguity rather than selecting the combination that produces the cleanest sentence.

Preserve uncertainty and attribution

Words such as may, estimated, preliminary, and proposed carry information. Removing them can strengthen a claim beyond the source. Similarly, changing one engineer suspects a cache issue into the cache caused the outage converts a hypothesis into a conclusion.

Create review examples that differ only in certainty or attribution. Ask reviewers to identify whether the summary preserves the distinction between an observation, an interpretation, and a decision. These cases are often more revealing than examples where the summary invents an obviously unrelated fact.

Keep disagreement visible when it affects the reader's decision. If two teams report conflicting dates, the summary should not average them or select one without a stated rule. A short note describing the conflict can be more useful than a polished but unsupported single timeline.

Exceptions deserve an explicit path into the output

A rule and its exception may appear far apart in a document. The model may retrieve the main rule and never see the qualification. Before summarization, ensure that the source preparation retains headings, footnotes, and references that connect these parts.

For the weekly briefing, a test may pass except on one supported browser. If that browser matters to the release, the exception belongs beside the result, not in an omitted appendix. Define importance according to the reader's task rather than the number of words the source devotes to a fact.

Ask reviewers to identify consequential omissions separately from invented claims. A summary can contain only true sentences and still mislead by leaving out the condition that changes their practical meaning. Faithfulness includes the relationship between what is retained and what is excluded.

Long inputs require a plan for aggregation

When a collection is summarized in stages, intermediate summaries become another representation that can lose details. A final summary of summaries may never receive the original exception or the precise source wording. Preserve evidence references and required fields across stages instead of passing only free-form prose.

Group documents by a meaningful unit, such as project or decision, rather than arbitrary file count. Keep cross-document relationships available. An action assigned in one meeting may be completed in a later issue update, and the final briefing should reflect the latest supported status under a clear ordering rule.

Retain the ability to return to original material for uncertain claims. A staged workflow should reduce the amount of text that needs close inspection, not sever the connection to the sources. If a final claim cannot be traced through the intermediate records, it deserves review before publication.

Evaluate claims, omissions, and reader effort

Prepare a set of documents with reviewed expected facts and known traps: conflicting dates, similar entity names, corrected numbers, and tentative decisions. Include ordinary cases too, so the evaluation reflects the real workload rather than only adversarial puzzles.

Assess each summary for supported claims, incorrect claims, missing consequential information, and readability. A single overall score can hide a tradeoff in which prose improves while accuracy declines. Keep the dimensions separate long enough to understand why reviewers preferred or rejected an output.

Measure the effort required to verify the summary. If reviewers must reread every source from beginning to end, the workflow may not save much time. Good source links, structured evidence, and clear uncertainty can make review faster without lowering the standard for accuracy.

Design correction and versioning into the workflow

Sources change. A project owner may correct an issue status after the briefing is drafted. Record which source versions produced each summary and decide when a change requires regeneration or a targeted correction. A summary without provenance can remain confidently wrong after the underlying record is fixed.

Avoid silently overwriting a published briefing when readers may have acted on it. Keep an appropriate correction record and make the revised meaning clear. The exact presentation depends on the product, but the internal system should preserve enough history to explain what changed and why.

Treat reviewer edits as evidence for future improvement, not automatically as new ground truth. A human edit can also introduce an error or reflect a preference unrelated to factual quality. Review the reason for the edit before using it to tune prompts or train another model.

Keep compression accountable to the source

A good summary helps the reader spend less time while retaining the facts that matter. It preserves quantities with their scope, decisions with their status, and uncertainty with its source. It also makes important omissions and unresolved conflicts visible enough that brevity does not become distortion.

Begin with a clear contract, retain an evidence trail, and evaluate the exact kinds of meaning your workflow cannot afford to lose. Different readers may need different summaries of the same documents. That is acceptable when the selection is deliberate and the retained claims remain faithful.

The strongest summarization system is not the one that always sounds most certain. It is the one that produces a useful shorter account and gives reviewers a practical way to verify, correct, and explain it.