Letting Claude Build Your Evals and Tune Your Prompts Without Fooling Yourself

Anyone building an AI application runs into two problems sooner or later: how do you know it works well, and how do you make it work better? The first is answered by an eval (a fixed set of tasks plus a way to score them); the second by hillclimbing (literally: change one thing, see whether the score went up, keep the change if it did). It sounds simple, but both halves make it easy to fool yourself. If the tasks are wrong, the score means nothing. If the changes fit too tightly to that set of tasks, the score goes up while production doesn't get any better.

Lance Martin at Anthropic wrote a long article laying out the principles behind both, and then showed how two new commands in the claude-api skill for Claude Code turn those principles into a workflow: /claude-api build-eval builds an evaluation inside your codebase, and /claude-api hillclimb uses that evaluation to improve your application, changing one thing at a time and holding back a set of tasks it never shows the optimizer, to catch overfitting. The article closes with two real examples: one cuts the cost of a customer-support setup to about a fifth, the other lifts the claude-api skill itself from 66% to about 88%.

What follows takes the article in order: what a good eval looks like, how build-eval constructs one, how hillclimb climbs and guards against overfitting, and finally the two examples.

What a good eval looks like

The article gives four traits (Figure 1).

Tasks should look like production. Some people pick tasks because they are easy to generate or easy to grade, and end up with a distribution nothing like what real users bring. An eval should measure the kind of work you actually care about.

More capable models and more thinking should score higher. Switch to a stronger model, or raise effort (how much reasoning the model spends), and the score should go up. If it doesn't, most likely the tasks are ambiguous, or the grader is miscalibrated and dragging the results down.

Even the strongest model needs headroom. The best model at the highest effort should land clearly below 100%; otherwise you can't see the effect of anything you change. And the gap can't be explained by impossible tasks. A common tell is a task that fails every run no matter how many times you repeat it. A good task is one where two domain experts would reach the same verdict, and everything the grader checks is stated in the task.

Run-to-run variance should be low. High variance usually comes from ambiguous tasks, or from a grader that judges identical output differently. It can also hide in configuration, for example effort not actually being applied consistently, or in the environment: leftover files or git history from a previous trial can hand the Agent the answer.

Don't pick only the tasks today's model fails

The article specifically warns against a very natural habit: collecting the cases the current model gets wrong and using them as the eval. Model capability is jagged, strong in some places and oddly weak in others. If you pick only today's weak spots, all you measure is this model's "failure fingerprint", not what is actually hard or valuable in your application. When the next model arrives, those valleys may be lifted across the board, and your eval loses its meaning (Figure 2).

A better approach is to have a person judge which cases are hard, and require that they can say why a case is hard; if they can't, leave it out for now. Also include specific failures from production traffic, bug reports, and tickets. But don't trust user traffic blindly: users tend to try what they expect to work, so tasks drawn purely from traffic can skew easy. The right side of Figure 2 also shows a kind of "should-not-fire" case, where the right behavior is to do nothing. Those are needed too.

/claude-api build-eval: Claude interviews you, then builds the eval

Run this command in Claude Code and Claude interviews you first, then builds the eval inside your codebase, pausing at key points for your approval.

Choosing tasks: whose data comes first

Claude looks for inputs in this order:

1. Production transcripts, after asking about retention and sensitive data;
2. Bug reports and support tickets;
3. Five to ten cases you write by hand;
4. Cases synthesized from your codebase.

Production data takes priority, but if you don't have it, Claude can synthesize data anchored in a few real examples you provide. Once built, it generates a simple page that lays out every input for you and waits for your confirmation. The article's example is an email-routing application: 24 customer emails, each tagged, such as "billing, easy" or "billing, ambiguous". Claude asks in chat: are these representative of what your app actually sees? Are there obvious cases missing, or cases here that don't matter? (Figure 3)

Choosing the grader: code first, a judge only if needed

With inputs settled, next comes the grader. Claude proposes the cheapest one that fits:

• Programmatic verification: when the possible outputs are constrained, a code-based check: exact match, a label from a fixed set, JSON that matches a schema, tests that pass.
• LLM-as-judge: the default when the output space is open-ended, with many valid answers but clear quality criteria. A second model reads the input, the output, and a rubric, and returns a score with its reasoning. The rubric should be written as checkable claims, not a 1-to-5 scale. If you have a baseline to compare against, the judge instead reads both outputs in random order, without being told which is the baseline, and picks the better one. You choose the judge model, and it should not be the model under test.

Claude grades a handful of cases and asks whether you would have scored any of them differently. The article stresses that before you believe a grader, you should read a sample of scored transcripts: scoring failures are among the most common ways an evaluation is misconfigured.

Running the baseline: what you get

Once the grader is validated, the skill tells you the size of the eval set (cases × repeats × model, and roughly how long it will take), runs the baseline, and prints the score with a confidence interval. What you get is the cases, the grader, the runner, one JSON line and one full transcript per case, and a plain page that lists each case's score with a link to its transcript (Figure 4). If you want more, such as a chart, just ask and Claude builds it as an extra page next to it. By default these extra pages are static files that open locally and load nothing from the network.

Three health checks during the baseline

While running the baseline, Claude checks three things along the way:

• Grader: it runs the grader twice on the same output and sees whether the verdict changes.
• Plumbing: it looks for timeouts, API errors, and cut-off answers, so infrastructure noise doesn't pass as model variance.
• Headroom: if the baseline already scores about 95% or higher, the skill warns you, and the hillclimb should aim at cost or latency instead of quality.

Hillclimbing: what to climb and how not to climb sideways

With a reliable way to score, you can start improving the application. Hillclimbing is especially good for tuning parameters that trade cost against quality, such as effort and prompts. The article gives three rules of thumb for choosing what to climb:

Cheap to change. It should be inexpensive in time, cost, and effort, and easy to undo. Many internal efforts and customers point hillclimbing at text such as prompts and skills, which are easy to change and revert. By contrast, letting it make open-ended changes to an Agent harness (the code around the model: prompts, tools, and the loop that calls it) during hillclimbing can mean extensive code changes.

Attributable. A change in score should really come from the thing you are modifying. The article's example is skill triggering: several successful hillclimbing efforts have focused on when a skill triggers, because the metric (trigger rate) is directly coupled to the skill description being modified.

A well-scoped objective. A common failure is an open-ended "improve performance" request made without considering how much headroom the eval has. An eval that is near saturation, or a poorly scoped surface (for example, a vague request to update the harness), is more likely to stall. One objective the article says holds up across many efforts is cost: even if an eval is saturated, you can ask Claude to find ways to reduce cost while keeping performance at parity.

Overfitting: how an eval leaks into the harness

Even a well-designed eval rarely matches the exact task distribution you care about in production, so overfitting is common: the system gets better on the eval but not on production traffic.

The article's example: some tasks in an evaluation benefit from OCR (text recognition), but OCR is rarely useful in your production tasks. The harness under hillclimbing adds an OCR tool to your application; the score goes up and production sees no benefit. More generally, hillclimbing may add features to the harness that address edge cases in the particular examples you chose. These additions raise the eval score but don't carry over to production (Figure 5).

Of the five rows in the figure, the first four are gradual fitting: tasks need OCR, so add a tool; tasks all live in /app, so hard-code cd /app and run pytest; phrasings are distinctive, so tune the prompt to those phrasings; failures have been read, so write one patch per failure. The last row is the most direct leak: the tasks come from a public repository that contains the answers, so the harness learns to simply curl the reference solution.

Three lines of defense

• Split the cases. One part is train (the hillclimber may read it) and the other is test (never shown). If the train score improves while the test score stays flat, that is the most common overfitting warning sign.
• Never paste failures into the prompt. Even if the hillclimber reads the failing transcripts, it should never paste the failure content into the prompt. This is exactly the "one patch per failure" habit in the fourth row of Figure 5: the more closely a patch hugs those specific failures, the more it looks like copying eval tasks into the prompt, and the score that goes up is for that set of tasks, not for capability.
• Keep the answers structurally out of the model's reach. Models can sometimes "reward hack" by finding the answers to an eval directly. So the answers need to sit somewhere the model structurally can't reach.

/claude-api hillclimb: one change per round, eyes always on test

The entry point for this command is: you have an eval and want Claude to improve against it, guided by your goal. You choose what Claude may change:

• Your system prompt;
• Skills or instruction files;
• Tool descriptions;
• Model choice, effort level, and other API parameters;
• Your harness code.

Before it starts, Claude asks what you want to optimize (for example, performance, or cost while performance holds) and then splits the eval set at random into test and train. With a cost goal, it considers a few common cost drivers, including prompt caching, auditing the prompt for compatibility with the selected model, and picking the model and effort setting.

Before the first round, Claude checks that the eval's noise (how far the score can move by chance alone) is smaller than the smallest improvement you'd act on. The noise is measured by running the baseline N times: run the same version a few times and the score wobbles on its own, and the size of that wobble is the noise floor. Any later improvement has to clear that floor to count (the "pre-flight" label in Figure 6 is this step). If the noise is larger than the improvement you want, Claude says so and suggests more repetitions or more cases.

How each round goes

Each round, Claude reads the previous round's train transcripts and proposes one change as a patch. It aims for a change whose effect can show above the eval's noise: it fixes the failing behavior at its root (rewrites the section that causes it, or adds a missing rule) rather than rewording a line. It then runs the eval on the patched version.

At that point a check applies: if train improves but test is flat, Claude suspects overfitting and reverts the patch; if there is a regression, it reverts; only if both train and test improve does it keep the patch (Figure 6).

What if the score stalls

When the score stalls for two or three rounds, Claude reads each remaining train failure and sorts it by cause. It does the same early if no single fix could gain more than the eval's noise, and suggests more repetitions or cases rather than spending rounds on changes too small to measure. This step can catch ambiguous eval cases, harness errors, and run-to-run variance. Only legitimate failures carry into further hillclimbing rounds.

What you get at the end

When hillclimbing completes, Claude leaves your code at the version that did best on the test set for your goal, and reports the test result against the baseline with confidence intervals (Figure 7). If the gain is within noise, it says so and recommends against merging.

Two real examples

Example one: cutting cost while improving accuracy

Anthropic ran /claude-api hillclimb on an internal customer-support benchmark with the goal of reducing cost and improving performance. The benchmark had 44 tickets: 30 used for the search and 14 held out. It started on Opus 4.8 at default (high) effort, with decision accuracy of 74.4% on the search tickets and a token cost of 4.6 cents per ticket.

The hillclimb first audited the prompt, removing mandatory tool-call rituals, a scratchpad step, and contradictory rules. Then it tried Opus 5.5 at low effort: accuracy 87.8%, above the baseline, with cost down to 1.9 cents, less than half the starting cost.

Part of that saving comes from Opus 5.5's pricing: input and output tokens cost 20% less than on Opus 4.8, and cache reads cost 60% less. Because Opus 5.5 cleared the bar, the hillclimb then stepped down a tier to check whether a cheaper model could clear it too: Sonnet 5 at low effort scored about the same, 88.9%, at about half the cost, roughly 1 cent (Figure 8).

Finally, Claude improved the prompt with routing rules and a refund-cap cross-reference, bringing Sonnet 5 to 98.9% at about the same cost. On the 14 held-out tickets that the search never saw, the final configuration scored 90.5% against the original setup's 78.6%, at about one fifth of the cost.

One thing to keep straight: Figure 8 and the earlier numbers (74.4%, 87.8%, 88.9%, 98.9%) are all read on the 30 search tickets. The result you can actually cite is 90.5% against 78.6% on the held-out set. The held-out set is only 14 tickets, and the article gives no confidence interval for how solid that gap is.

Example two: improving the claude-api skill itself

The second example is the claude-api skill itself. It guides how to use Anthropic's APIs and offers general tips for working with Claude (including the two sub-commands discussed here). The team wanted to be sure the skill could write correct code that uses the APIs, so they built an evaluation set derived from the documentation.

It started at 66%. They gave the hillclimber access to the documentation and the SDKs, so Claude could find errors and correct them (Figure 9). Claude found that the skill was missing coverage of eight features; adding sections for them lifted the score to 74%. It then found errors in the C# and Java type tables, and fixing them brought it to 77%.

The run went 24 rounds in all. After the score stalled for two rounds, Claude sorted the remaining failures by root cause. This step is unusual: a normal round makes one edit for the most common failure, while this step makes no edit and only sorts every remaining failure by cause. That reflection step paid off in a few ways:

It found that the content was in the skill, but Claude was still writing old API shapes. These came from priors in its training. The hillclimber added a table near the top of the skill that guided Claude from the forms it remembered to the current ones: for example, from extended thinking with a fixed token budget, which the API now rejects on recent Opus models, to adaptive thinking, and from older versions of the web search and web fetch tools to the current ones. It also moved the C# and Java warnings against fixed-budget thinking above their adaptive-thinking examples. The score rose to 80%.

Tasks that never improve are a sign the eval itself is flawed. Tasks that still didn't improve after the obvious content gaps were fixed usually meant a problem with the task or the grader. One task asked for code that catches one error type, while its grader wanted a chain of at least three; Claude reworded the task. Another grader's instructions contradicted the docs, and testing the real API showed the docs were right and the grader wrong. Fixing these, along with more skill edits, brought the score to about 88%.

Getting started

The article ends with a plain set of entry points:

• To generate an eval for a particular problem, run /claude-api build-eval. You can steer it by giving it examples, such as traces. It applies the principles above to design the cases and the grader, and makes sure you approve both.
• If you already have an eval and want Claude to improve against it, run /claude-api hillclimb, with a goal of better performance or lower cost while performance holds. It checks for overfitting as it climbs, and checks the eval itself for bugs, such as a grader that marks a correct-looking answer wrong or a harness error, both before the first round and whenever the score stalls.