I'm a Solo Lawyer in Austin. My Citation Checker Posted the Highest Score on Princeton's AI Benchmark.

Why I Built Verity

I'm a solo lawyer in Austin. My practice covers transportation, corporate law and litigation. I am not an AI researcher. I do build AI agents, more than 100 of them by now, most through several versions, because I have never been satisfied with what an off-the-shelf model produces.

I run my practice with a team of them, each with a name and a job. Karen is my chief of staff. Callie keeps the docket. Travis runs quality control on everything that leaves the office. Covenant checks contracts before anyone signs. And Verity checks sources: every citation in what we file, and every citation in what the other side files.

Citations were the part I trusted least. Courts keep sanctioning lawyers for filing briefs with cases that don't exist or quotes that were never written. I went through 115 of those decisions, and judges imposed sanctions in about half of them. A hallucinated citation is a misrepresentation to the court, and that is what gets a lawyer sanctioned. I was not going to let AI near anything I file until I had built the best checker I could. That checker is Verity.

I thought she was good. But every builder thinks that. I wanted to find out on someone else's test.

Princeton Built a Racetrack

This year, researchers at Princeton published LePhantomCite, a benchmark for exactly this problem.¹ It has 390 excerpts, 300 of them from real federal appellate briefs, carrying 331 citation errors: fake cases, case names attached to the wrong citation, wrong page numbers, altered quotations, and real cases cited for things they don't say. The researchers ran GPT-5, Gemini, several open models, and Anthropic's own Claude Code, and published every score.

I didn't build Verity for this test. I didn't know it existed until Claude found it for me while I was planning a benchmark of my own. She was built to check filings in my practice, and her instructions were last edited the day before I first heard of LePhantomCite. I didn't change a word of them for Princeton.

They built the track and posted the lap times. I drove up in a car I built in my garage, on the same engine as their fastest car, Claude Opus 4.8. Verity normally runs on the newer Opus 5.5, but I put the older engine in on purpose, so nobody could say I won with a better motor. Before the run, I froze Verity and recorded a fingerprint of every file, so nothing could change mid-race. Then I ran her once, timed by Princeton's own scorer, and kept every result.

She also raced with part of her toolkit switched off. To keep the run clean, the test setup blocked the scripts she normally runs on every citation, so she worked from her own judgment and her case-law lookup tools alone. The setup around her was new: a short task prompt that defined Princeton's five error types and asked for answers in their format, checked on 20 of their practice excerpts before the freeze. Every system that ran through my setup got that same prompt.

The Lap

For a lawyer, what matters is accuracy: not walking a bad citation into court, and not accusing the other side over a good one. Verity caught every fake case on the test, all 31.

GPT-5 caught about 38 more errors than Verity, and raised about 300 more false alarms to do it. More than half of its accusations were wrong; you would spend your night disproving them, or repeat one to a judge. Claude Code was careful but missed about 33 more real errors than Verity. Verity is the only system above 70 percent on both, and that is why she has the highest score.

Then I Checked Their Clock

Princeton's scorer said 93 of Verity's flags were false accusations. I wanted to know whether any of them were actually right, so I checked the strongest candidates on Westlaw, one opinion at a time, and kept only what held up beyond any doubt.

At least six held up. Every one is a citation error the answer key does not mark, and Verity flagged all six. One brief quotes the Supreme Court in Schware v. Board of Bar Examiners as saying "The practice of law cannot be licensed by any state." The opinion says no such thing; it says the opposite. Another quotes the Second Circuit's intent standard with the word "negligence" quietly swapped out.

That isn't a knock on Princeton. Building a benchmark is hard, theirs is good, and that's why I used it. It does mean the official score counts some of Verity's best catches against her.

Her Name Isn't Her Engine

I also ran a control: the same model, tools and task prompt, but without Verity's own instruction file, which carries her persona: her role at my firm, who she reports to and how she hands off work. Her persona cost about two-thirds more to run ($1.15 per excerpt against $0.69) and made no measurable difference to her score (72.5 against 72.2).

That's not a knock on the persona. This test asks only whether she can spot a bad citation. My firm needs more than that: the right finding, to the right person, in the right form, before anything goes to a court. The test measured her engine. The persona is how she does her job. The lesson I'm taking: keep them separate, and load each one when it's needed.

What This Does and Doesn't Mean

It doesn't mean Verity is the best citation checker in the world. The overall margin is real but narrow, and she still misses about a quarter of the planted errors, most often misstated holdings and wrong page numbers. The errors were planted; real filings are messier. This is one run, and I have reported all of it.

Here is what I think it does mean. Verity ran on the same model as the fastest system Princeton tested. The difference was the build: decisions about how a lawyer actually checks a citation. Open the real opinion, not a search snippet. Quote the passage, not a summary of it. Define exactly what counts as an error. None of that came from a lab. It came from practice.

That's the part I want other lawyers to hear. The people closest to the work can build tools that hold up against the best general-purpose systems in the world, if they're willing to test them honestly.

Back to the Garage

The track showed me exactly where Verity is weak, so the next version fixes it: her scripted checks run on every citation, every time, outside the model's discretion; page-level checking for pinpoint citations; a structured reading of every holding; and a rule that every flag must quote the passage that contradicts the brief. It will be frozen and tested on held-out data before I report a new number. I am also building a benchmark of my own, from real sanctions orders, to test the harder question this one leaves open.

Notes

1. Patty Liu, Dominik Stammbach and Peter Henderson, Who Checks the Citations? Benchmarking Legal Hallucination Detection, arXiv:2606.21155v3 (2026). arxiv.org/abs/2606.21155