Every document grows from a 1,000-token script

Writing this many files by hand was never an option. Harvey's approach: for each matter, first write a "script" of roughly 1,000 tokens that covers two things — which client this is for, and roughly what the project looks like. Then the script is stuffed with concrete facts: some are fine-grained, like "the escrow amount in this agreement is 10%" or "the non-compete is two years"; others are structural, like "this litigation was dismissed" or "this one settled."

The clever part is the last step: to define the ground truth, you only need to read that 1,000-token script — you don't have to review the millions of rendered documents. Otherwise, who's going to audit 108 million tokens?

Those planted fact points have a second identity: they are the scoring answer key. But the AI never sees them. It only gets the stack of Word documents and has to use search and reasoning to dig up scattered clues and piece together an answer.

The rendered file system looks like this: matter number on the left, that matter's own folders in the middle (due diligence, financing, insurance, regulatory, closing docs...), and the actual .docx files on the right. Image: Harvey

How real does the synthetic data get? I downloaded one to check: second-request-strategy-memo.docx, 66KB, 344 paragraphs, a fully fleshed-out lawyer's memo. The header says "Confidential · Attorney Work Product." It cites 15 U.S.C. § 18a, notes the 2024 filing threshold of $119.5 million and the $160,000 filing fee, and walks through a timeline — purchase agreement signed June 4, filed June 18, regulator requested materials July 8, formal second request issued July 16 — with zero inconsistencies across names, companies, and dates.

The initial 30-day statutory waiting period under the HSR Act was set to expire on July 18, 2024. However, on July 8, 2024, the FTC's Bureau of Competition, Mergers IV Division issued a voluntary access letter … We produced 23 supplemental documents on July 10, 2024, in response to that voluntary access letter.

All 250 questions sound like a partner's offhand remark

So we have the archive. What do the questions look like? The 250 questions simulate real requests a law firm might make to its own document management system. They're plain-spoken, single-sentence asks — the kind of thing a partner blurts out while walking past your desk in the morning.

Official sample tasks. The original wording on the left, the task category on the right: find precedents, take inventory, benchmark, find exceptions, spot trends, cross-practice client checks... These are questions firms ask every day. Image: Harvey

I pulled the actual task files from the repo. Question 1 reads: "I'm reviewing the transactions that triggered regulatory scrutiny in my antitrust group. Pull all the matters that received an HSR Second Request, along with the related correspondence."

The ground truth for this question has 11 items:

Notice that last rule: even the test authors had to write a paragraph explaining which matters count and which don't.

But there's no index. The AI has to read every file.

One more thing to be clear about: for every single question, the AI faces the entire file system. All 266 matters are spread out in front of it, with no one telling it where to look.

The agent has six tools: run command, read file, write file, edit file, find by filename, and search by content. It does get grep — the problem is having a keyword to search for.

I checked the file system myself: under dms/matters/ there are just 266 numeric directories. No index. No client list. No practice-area mapping. No metadata file of any kind. To find out what client 1003 is, which practice area it belongs to, or how the matter resolved, you have to open the Word documents and read them one by one.

And the rule is: miss one checkpoint, get zero

Each question's ground truth is split into individual pass/fail checkpoints, judged one by one by an AI grader (claude-sonnet-4-6, temperature set to 0, looking for semantic equivalence, not keyword matches). All checkpoints must pass to earn 1 point—miss just one and you get 0. There's no partial credit. Harvey calls this all-pass scoring.

Why be so strict? The reasoning goes:

In real legal delivery, averages are misleading. A due diligence memo that catches 95% of the issues but misses one material one isn't "95% useful." It's wrong. Harvey LAB eval methodology doc

That missed issue could change the deal price, force a redo before closing, or blow up after signing.

Neither frontier model passed

Rules out of the way, here are the scores. Harvey ran two models through LAB's standard execution framework:

Official baseline scores. Top row is accuracy, bottom row is time and money. Image: Harvey

An individual checkpoint pass rate of 70–80% sounds decent. But when the bar shifts to "answer the whole question without missing a single point," it collapses to 20–30%. 475 seconds is about eight minutes—per question. At median cost, running all 250 questions once comes to roughly $300. The two models mostly overlap on the easy wins, and each manages to crack a few the other can't.

Checkpoints stay steady. Full-pass collapses.

Looking at the overall score isn't enough. Split the 250 questions into four tiers by "how many checkpoints the ground truth has," and the two lines head in opposite directions.

The other model, gpt-5.6-sol, holds up a bit longer—full-pass rate goes from roughly 51% to 27% to around 20%—but in the "26+" tier, it also hits exactly 0%.

Official chart with both models. The four points on the left could almost be a horizontal line; on the right, both curves dive together. Image: Harvey

The real problem: the AI doesn't know when it's done searching

So where exactly does it go wrong? Harvey dug into the agents' full execution traces, and the conclusion runs counter to intuition.

This is especially deadly for "give me the full list" style questions — which is exactly where the cliff in the chart above is steepest.

Harvey's root-cause conclusion: the agent never builds an internal model of "what's in this archive." Without that, it has no basis for judging whether its search is thorough enough.

Harvey's bet: read the archive into an index first

Legal work is fundamentally about figuring out how the current problem relates to what's been done before: which precedent applies to this client, what's market practice. But today's agents re-derive all that knowledge from scratch for every single task. At enterprise scale, that approach is both too expensive and too weak.

That's the direction they're betting on: have the agent first read the archive into its own index, then answer questions. Because the environment is persistent — the same archive gets asked 250 times — the one-time cost of "getting to know" the corpus pays off. Expect more releases in this vein; their partner Engram literally builds the memory layer for AI, having emerged from stealth this June with $98M in funding.

Harvey is also explicit about the dataset's limits: it currently covers only part of law firm work, and 250 questions are just a subset of what lawyers might ask. More will be added. Also, looking at the run logs, both cost and time scale up with corpus size — and a real enterprise archive can be orders of magnitude larger than this fake firm.

The AI gets search tools, but there's nothing to search *for*. The repo contains 266 numeric folders, no index, no client list, no explanations. To learn whose matter it is or whether it belongs to a certain group, you have to open documents and read.

Even the test authors had to add a note in the answer key explaining which matters count and which don't.

The answer key is split into individual checkpoints, judged by an AI grader one by one: all must pass for the question to count. Miss one, and it's a zero — no partial credit.

Harvey ran two frontier models: opus-4.8-high and gpt-5.6-sol-high. On individual checkpoints, both land in the 70–80% range. Require them to get *everything*, and they fall to 20–30%.

Scores come from Harvey's self-run LAB eval; no third-party replication yet. A single question takes ~400+ seconds and $1–2 at the median.

Harvey reviewed the agent traces: the models' reasoning about what they found is mostly right. What fails is completeness. At the core, they have no model of "how much is even in this archive," so they can't judge whether they've searched enough.

The fix: have the AI spend the effort once to read the archive into an index, summary, and memory — then answer questions. Same 250 questions hit the same files, so the upfront cost amortizes.