Harvey open-sources a synthetic law firm with 108M tokens and 9,288 documents
- Harvey built a fake law firm from scratch, put all files for 266 matters on GitHub, and let AI agents dig for answers.
- Two frontier models ace about 70–80% of individual checkpoints, but when asked to get every single one, they drop to 24.4% and 36.0%.
- Questions with more than 16 checkpoints see the full-pass rate fall to exactly zero — while the models don't actually get dumber on hard questions.
The firm is fake. The scale is real.
Legal AI company Harvey has open-sourced an entire law firm's archive: 266 matters, 9,288 Word documents, and 108 million tokens are all on GitHub, paired with 250 questions for AI to answer. The archive, the questions, the scoring rubric, and the benchmark results are all publicly available.
Why build this dataset? In a real law firm, finding things is a pain: documents pile up endlessly, and much of the key information has no fixed keyword you can search for. Previous legal benchmarks were all "here's a contract, answer a question" — the material was handed to you, so they never tested this skill. This dataset is a new extension of LAB, Harvey's legal agent benchmark.
First, you need to understand how a law firm organizes its work, because that's exactly how this dataset is built. Everything runs on two relationships: the client (who you're working for) and the matter (the specific piece of work for that client). Here, 46 clients span 15 practice areas. One client can have many matters, and one matter might pull in lawyers from several practice groups.
The 46 clients include both companies and individuals, and they were deliberately made as diverse as possible: what a PE fund needs is completely different from what an industrial manufacturer needs. The more varied the clients, the more practice areas you can cover. Of the 266 matters, some are closed and some are still active.
It's one specific piece of work a firm does for a client — everything from the engagement letter to the closing memo lives under that matter. Think of it like a "project" folder in project management software, except in a law firm it's also the billing unit and a drawer in the filing cabinet. The folder number is "client number, hyphen, sequence number": 1003-00001 is the first matter for client 1003.
A matter's documents cover its full lifecycle: how the work was won, how it was staffed, the major decisions along the way, and the final outcome. Files are organized into folders per matter, but the organization is deliberately inconsistent — each matter's folders look different, depending on the matter type, the partner's habits, and how the work actually played out. Real firms are just this messy; nobody standardized how files should be arranged.
Every document grows from a 1,000-token script
Writing this many files by hand was never an option. Harvey's approach: for each matter, first write a "script" of roughly 1,000 tokens that covers two things — which client this is for, and roughly what the project looks like. Then the script is stuffed with concrete facts: some are fine-grained, like "the escrow amount in this agreement is 10%" or "the non-compete is two years"; others are structural, like "this litigation was dismissed" or "this one settled."