A brief is filed citing six decisions in support of a motion to dismiss. Two of the decisions do not exist, a third exists but says nothing like what the brief attributes to it, and the attorney who signed the brief learns this only when opposing counsel's reply points it out. Courts in several jurisdictions have confronted versions of this scenario, and some have sanctioned the lawyers involved [1]. The usual response inside firms has been to remind lawyers to check what the software gives them. That instruction is correct as far as it goes, but it treats verification as a final step performed by a tired reviewer. The more durable answer is to build verification into the systems that produce the draft in the first place.
Hallucination follows from how generative models produce text
Large language models generate text by predicting, one fragment at a time, what is likely to come next given their training data and the prompt in front of them. Nothing in that process consults a reporter, a docket, or the record of a case. To the model, a citation is a familiar pattern of volume number, reporter abbreviation, and page. It can produce a well-formed citation to a case that was never decided as easily as it can reproduce a real one. The output reads fluently because fluency is what the model is built to produce, and accuracy follows only when the training data and the prompt happen to align with the truth.
Fabricated citations are the most visible failure, though not the most dangerous one. Anyone who runs a nonexistent case through a research service will catch it, and opposing counsel usually will if the drafter does not. Subtler errors are harder to detect: the real case cited for a proposition it does not support, the dictum presented as a holding, or the paraphrase placed inside quotation marks. Factual work carries the same risk, since a model asked to summarize a deposition transcript can attribute to a witness a concession the witness never made. Each of these errors passes a superficial existence check, and each can find its way into a filing.
The duty to verify rests with the signing lawyer regardless of the tool
The governing standards did not need amendment to reach this problem. Under Fed. R. Civ. P. 11(b), an attorney who presents a filing certifies, after an inquiry reasonable under the circumstances, that its legal contentions are warranted by existing law or by a nonfrivolous argument for changing it, and that its factual contentions have evidentiary support [2]. Rule 26(g) imposes a parallel certification on disclosures and on discovery requests, responses, and objections. Neither rule asks how a draft was generated. A citation the lawyer never read is not the product of a reasonable inquiry, whether it came from a model, a summer associate, or an old form file.
The Model Rules point in the same direction, with different emphasis. ABA Model Rule 1.1 requires competence, and Comment 8 extends that duty to understanding the benefits and risks of relevant technology. Model Rule 3.3, by contrast, prohibits knowingly making a false statement of law to a tribunal, so an unwitting fabrication may fall outside it even though Rule 11's objective standard does not. Supervision is the concern of Model Rules 5.1 and 5.3, which make lawyers responsible for the work of those they direct, and ethics authorities have read the latter to reach outside technology providers [3]. The ABA has also issued a formal opinion on lawyers' use of generative AI tools [4][5], and some federal judges have adopted standing orders requiring disclosure or certification of AI use in filings, on terms that vary considerably from court to court [citation needed].
Review at the end of the process fails in predictable ways
If the duty is clear, it is fair to ask why lawyers who understand it still file fabricated authority. Part of the answer is that end-stage review asks people to do what they do worst, which is to find errors in fluent, confident text that looks like competent work. Research on automation bias suggests that people tend to scrutinize the output of automated systems less closely than they would a colleague's work [6]. Deadline pressure compounds the effect. When a filing runs late, the review step is the first thing to be compressed.
An illustrative matter shows how this happens. In Meridian Analytics, Inc. v. Halvorsen Data Partners, LLC, a hypothetical trade-secret dispute in federal court, Halvorsen's counsel uses a generative tool to draft an opposition to Meridian's motion to compel production of source code, and the draft cites 11 authorities on proportionality under Rule 26(b)(1). Working the night before the filing, a second-year associate confirms that each case exists and remains good law. Because time is short, she skips the pin cites and so never discovers that two of the decisions addressed relevance objections, not proportionality, or that one quoted passage appears nowhere in its opinion. Her review was genuine and still inadequate, because it tested the property the system made easy to check instead of the properties that mattered.
Verification architecture begins with traceability to sources
A system built for legal work should make every assertion traceable to the material that supports it. The most common approach, usually called retrieval-augmented generation, restricts the model to a defined body of documents (a research database, the case file, a production set) and requires it to identify the passage each statement relies on [7]. Grounding of this kind reduces fabrication without eliminating it, since a model can still misread a retrieved passage or attach a citation to a sentence the passage does not support [8]. Traceability is therefore less a guarantee of accuracy than a way of making the lawyer's review efficient and complete.
The practical test is granularity. A tool that lists its sources at the end of a memorandum leaves the reviewer to reconstruct which source supports which sentence, which is nearly the same work as checking the draft from scratch. Linking each proposition to a specific page or paragraph changes the task: the reviewer can open the source beside the claim and decide whether it holds. Factual work calls for the same discipline, with a deposition summary pointing to page and line and a chronology built from a production citing Bates numbers. Where the system cannot locate support, the better design declines to state the proposition and says so, instead of filling the gap with something plausible.
Deterministic controls catch errors that probabilistic systems miss
Traceability depends on the model behaving well, and deterministic controls do not. A deterministic check returns the same result every time it runs on the same input, and its logic can be inspected and audited, which makes it a natural complement to a generative model whose output varies from run to run. Such checks are narrow by design. They cannot tell a lawyer whether an argument will persuade, but they can establish mechanical facts about a draft that no reviewer should have to establish by hand. A well-designed system runs checks of the following kind before a draft reaches the lawyer:
Each citation is parsed and matched against an authoritative database, and any citation that cannot be matched is flagged as unverified.
Every quotation is compared character by character against the text of the cited source, with any discrepancy displayed to the reviewer.
Pin cites are confirmed to fall within the page range of the cited opinion.
Subsequent history and negative treatment are retrieved for each authority from a citator.
Record citations, such as transcript page and line or Bates numbers, are confirmed to exist in the case file.
None of these steps is new, since cite-checkers and paralegals have performed them for decades. The difference is that the checks run automatically on every draft when it is created, instead of depending on whether anyone had time before the deadline. In the Meridian hypothetical, the quotation check alone would have caught the passage that appears nowhere in the opinion. A lawyer would still have needed to read the cases to catch the mischaracterized holdings. That division of labor is the purpose of deterministic controls, which clear away mechanical errors so that human attention goes to the judgments only a lawyer can make.
Accountability has to be designed into the workflow
Systems can reduce error, but they cannot hold a license or sign a filing. Firms that adopt generative tools therefore need to decide, in writing, who verifies what and how that verification is recorded. A workable policy identifies the categories of work product that require review of every cited source, assigns that review to a named lawyer, and preserves a record that the sources were opened and the automated checks cleared. Such a record helps supervising lawyers meet their obligations under Model Rule 5.1. It also becomes valuable if a court ever asks what inquiry preceded a filing.
For firm leaders evaluating tools, the same analysis supplies the questions to put to a vendor. The first is whether each output links to a specific source passage or only to a list of sources at the end. Others follow from the distinctions drawn above: which checks are deterministic and which depend on the model, what the system does when it cannot find support, and whether verification steps are logged. A vendor that cannot answer concretely is asking the firm to accept fluency as a proxy for accuracy. Still unsettled is what courts will treat as a reasonable inquiry when a lawyer relied on a well-designed verification system and an error got through anyway. Firms that can document their process will be better placed to meet that standard once courts define it.
References
Mata v. Avianca, Inc., No. 1:22-cv-01461 (S.D.N.Y. June 22, 2023), ECF No. 54 (opinion and order on sanctions). Search results gave the reporter cite 678 F. Supp. 3d 443, but the page I read did not show it, so check it before using it. https://law.justia.com/cases/federal/district-courts/new-york/nysdce/1:2022cv01461/575368/54/
Fed. R. Civ. P. 11(b)(2)–(3); Fed. R. Civ. P. 11 advisory committee's note to 1993 amendment. https://www.law.cornell.edu/rules/frcp/rule_11
Model Rules of Pro. Conduct r. 5.3 cmt. [3] (Am. Bar Ass'n). Check against the ABA's own text before publishing. https://www.courts.nh.gov/new-hampshire-rules-professional-conduct/rule-53-responsibilities-regarding-nonlawyer-assistance
ABA Comm. on Ethics & Pro. Resp., Formal Op. 512 (2024) (Generative Artificial Intelligence Tools). Get the opinion PDF from americanbar.org and cite it directly; I could not fetch it. https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-october/aba-ethics-opinion-generative-ai-offers-useful-framework/
Standing Order Establishing Disclosure and Certification Requirements for Generative Artificial Intelligence (D.N.M.) (undated) (applies to cases assigned to one district judge). https://www.nmd.uscourts.gov/sites/nmd/files/Standing%20Order%20Regarding%20Use%20of%20Generative%20AI.pdf
Raja Parasuraman & Dietrich H. Manzey, Complacency and Bias in Human Use of Automation: An Attentional Integration, 52 Hum. Factors 381 (2010). Link to the publisher's DOI (10.1177/0018720810376055) instead of ResearchGate. https://www.researchgate.net/publication/47792928_Complacency_and_Bias_in_Human_Use_of_Automation_An_Attentional_Integration
Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401v4, 2021). https://arxiv.org/abs/2005.11401
Varun Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (arXiv:2405.20362, 2024). Search results show a published version in 22 J. Empirical Legal Stud. 216 (2025). I did not read that version, so check it and prefer it if it matches. https://arxiv.org/pdf/2405.20362
