Why Lawyers Cite Imaginary Cases and What It Says About the Way We’re Using AI

A few months ago, a US court had to deal with a surreal problem: a legal brief that confidently cited a series of cases that didn’t exist. The lawyer hadn’t meant to deceive anyone. They’d asked a generative AI model to find relevant precedents and, under pressure, trusted what it produced. The model obliged with fluent nonsense.

How does something like that still happen? Probably because the lawyer wasn’t using an integrated system at all. It was almost certainly what we now call shadow AI: the quiet, unsanctioned use of public chatbots for professional tasks. 

The legal firm hadn’t built a safe, auditable environment, so the individual improvised at their desk or even at home. The result was predictable.

There’s a simple way to prevent this. In engineering, we call it redundancy; in aviation, fly-by-wire. Imagine running two independent large language models, say Claude and GPT, on the same prompt, and then having a third model, such as Gemini, adjudicate between them. 

If the first two disagree, or if either produces something that doesn’t appear in real case law, the adjudicator flags it for review. The chance of both models hallucinating the same fiction is vanishingly small.

That’s not exotic AI safety research. It’s the basic design discipline that keeps planes in the sky and data centres online: independent verification and exception handling.

The real issue isn’t intelligence; it’s integration. When organisations don’t provide safe, embedded tools, people reach for whatever’s at hand and the results end up in front of judges.

So a question for anyone deploying generative AI at work:

Are your systems architected for resilience and cross-check, or are you still relying on single-model trust and shadow AI?

Are you using AI at home and mailing it in?