Hallucination
Definition
A hallucination is a confident but false output from a language model — an invented figure, clause or reference. It is a property of statistical text generation, which is why production systems verify outputs against source data instead of trusting them.
The engineering answer is containment: retrieve facts from systems of record, force structured output, validate values against the database, and require citations for any claim.
Where a wrong value would be expensive — payments, contracts, statutory filings — a human confirmation step stays in the workflow by design.
In practice
- Mitigated by retrieval, validation rules and structured output
- Never eliminated entirely — critical steps keep human sign-off
- Log every output so errors can be traced and corrected
Related terms
- Retrieval-augmented generation (RAG)Retrieval-augmented generation is a method in which relevant company documents are searched first and the retrieved passages are given to the language model as context. The model answers from those passages, which makes answers current, source-linked and auditable.
- GuardrailsGuardrails are the technical limits around an AI system: allowed tools, value ranges, spending and volume caps, forbidden actions and mandatory approvals. They turn an unpredictable model into a system that fails safely and visibly.
- Human in the loopHuman in the loop means a defined point at which a person reviews, approves or corrects the AI system's work. The handover is designed in advance for specific triggers — value thresholds, low confidence, unknown counterparties — rather than left to chance.