Check Whether the Eval Framework Is Fair Before Judging Which Model Is Stronger

Before comparing two models' scores head-on, you have to ask one thing first: is the eval framework itself fair? Ploy ran the same eval suite across both model families, and the most surprising finding was that the framework had already been tuned around the old model — without the team realizing it.

On the first cross-model run, the bias showed up in a few specific places:

• The tool-call budget was set around Opus's sequential calling style. GPT-5.6 fans out a bunch of calls in parallel, blowing through the budget even on tasks it actually got right.
• The eval runner didn't support batch file reads — a move Opus rarely used but GPT-5.6 used constantly.
• One dataset was missing a minScore threshold and silently inherited the default value of 1.0. So a GPT-5.6 hero section that scored 0.98 got marked "failed," and an Opus task that passed every individual check also counted as a "failure." Both were defensible design directions — what tripped them up was an invisible threshold.

Tracing this round's raw failures back, about a third of them turned out to be caused by the eval framework's assumptions, not actual model behavior — and the two models didn't trip over them equally.

Left: Opus's gradient hero scores a perfect 1.0 and passes. Right: GPT-5.6's color-block approach scores 0.98, but gets marked as failed simply because it hit an unwritten hidden threshold that defaulted to 1.0. Source: Ploy Blog

The takeaway is direct: when evaluating a challenger model, read through the traces line by line before trusting the pass rate — otherwise you're grading the new model on "how much it resembles the old one."

Design Style: Clean and Attractive, but Prone to Sameness

Ploy's overall read on GPT-5.6's design ability: it's very good at clean, modern layouts with tidy grids, but left unguided, it converges toward that look every time. Under the old eval framework built for Opus 4.8, GPT-5.6 tended to ignore existing design systems and produce pages that were crisp and restrained, but unmistakably generic at a glance.

Left: Opus replicated Clay's visual brand system. Right: under an eval framework designed for the old model, GPT-5.6 produced a clean but generic page. Source: Ploy Blog

How to guide it toward world-class brand fidelity is a topic for another post. This is just noting a real capability trait — not really a pitfall.

The Model Invents a Fake Value for Parameters It Doesn't Need

Ploy's code tool has 25 top-level parameters, of which only action is required — the rest are optional. Claude sends only the two or three it actually needs and omits the rest. GPT-5.6 fills in all 25 every single time, inventing a plausible-looking value for whatever it doesn't need: offset gets 0, timeout gets 120000, siteId gets a UUID that's all zeros.

Over three days of production traffic, the share of code(read) calls carrying all 25 parameters was different enough that it barely looks like the same tool:

In absolute numbers: all 6,635 of GPT-5.6's calls came fully loaded; only 4 of Opus 4.8's 2,898 calls did; and none of Sonnet 5's 1,933 calls did.

Prompting can't fix this. Ploy tried every way it could think of to "talk it out of it," and all of them failed outright:

This is a behavior baked into how the model generates function calls at a fundamental level — instructions can't talk it out of it, so the only option is to design around it.

The fix that actually worked was a schema transformation at the provider boundary: only for OpenAI-family models, rewrite every optional property as "required but nullable" (using anyOf: [T, null]), so the model has a proper way to explicitly say "I'm not using this." Then, at the one checkpoint every tool call passes through, strip those nulls back out — the tool side doesn't need a single line changed. The net effect: the model sees a schema that lets it be honest, and the tool still gets the same input it always got.

// Before: 25 keys, every one carrying an invented value
{ "action": "read", "file_paths": [...], "offset": 0, "timeout": 120000, ... }

// After: 25 keys, 4 real values, 21 explicit nulls (stripped before the tool runs)
{ "action": "read", "file_paths": [...], "offset": null, "timeout": null, ... }

The Two Providers' Prompt Caching Are Fundamentally Different Designs

On the surface both providers call it prompt caching, but underneath that one word "caching" sit two completely different architectures. This is the single thing in the migration that needed the most care: before it was set up correctly, GPT-5.6 looked about 50% more expensive than Opus — and that extra cost came entirely from Ploy's own cache configuration, nothing to do with model pricing.

Ploy's prompt starts with a roughly 29K-token static prefix (tool schemas plus the core system prompt) that's identical in every single conversation. How the two providers cache that prefix is worlds apart:

OpenAI's official mechanism is explicit: mark with prompt_cache_breakpoint, paired with a required prompt_cache_key. The key is exactly where the design really forks, because it's part of the cache's identity. Same prompt, different key, and the hit rate is 0. Each key maps to one cache node, and that node can handle roughly only 15 requests per minute — go over that, and OpenAI routes the overflow to other, cold cache nodes.

So "whether to enable caching" turns into a real design decision: what level do you bind this key to? The three approaches each play out differently:

What Ploy ultimately shipped was a workspace-level key: requests get hashed to a throughput-limited cache node, which effectively gives every customer workspace its own dedicated lane:

The key alone wasn't enough — Ploy also split the system prompt into three tiers of breakpoints, each handling one job, aligned with the structure it uses for Anthropic:

After the fix, GPT-5.6's per-suite eval cost dropped below Opus's. Every dollar of the price gap the team had been staring at turned out to be a misconfigured cache — nothing to do with model pricing.

There's also one structural limitation that can't be worked around: on OpenAI, sharing that static prefix across workspaces just isn't structurally possible. Anthropic can share it because its cache is partitioned by organization, not by key. On GPT-5.6, once a given workspace's cache sits idle and expires, the next call has to pay for a cold 29K write again — about $0.18. That's a real cost, but it's bounded and predictable.

The Previous Turn's Reasoning Record Can Suddenly Go Missing — One Flag Fixes It

The last pitfall is the shortest one, but it can genuinely break a conversation. By default, GPT-5.6's Responses API stores the previous turn's reasoning as a "server-side item reference": the server keeps a numbered record, and the next turn just references that number. Ploy's conversations started intermittently throwing mid-conversation errors: Item 'rs_...' not found.

store: false — tells the SDK to request "encrypted reasoning content" instead, turning reasoning replay into a self-contained data blob
rather than a pointer reference to server-side state; otherwise it can throw Item 'rs_...' not found mid-conversation

One more thing cost Ploy an extra afternoon of debugging: once the round-trip carries server-side reasoning state, the effective upstream prompt can still change — even if the bytes you're sending strictly only append, with not a single one touched.

After the Migration: The Final Scorecard and a Checklist for Similar Teams

With all four pitfalls patched one by one, the numbers promised up front held up: about 2.2x faster, 27% cheaper, visual score flat-to-slightly-up; after reconfiguring the cache, first-hit rate went from 0% to 83.7%, missed-cache input tokens dropped 28%, and per-suite cost fell below Opus; after the schema fix, empty file reads hit zero and tool calls dropped by about 30%.

These four steps have a required order — fix the framework first, or you can't trust any of the numbers that follow:

Same agent, same dev toolkit — only the default brain changed. In Ploy's own benchmark, having the agent rebuild a brand homepage to match a reference design, on average:

The team also found GPT-5.6 has a quirk: left unguided, its designs lean clean but generic, and it tends to ignore the existing brand style (this is a capability trait, covered more in the full article). But what actually breaks results are four other invisible pitfalls. The sneakiest one: it quietly reads empty files and thinks it succeeded.

GPT-5.6 has a habit: every time it calls a tool, it fills in every parameter field, and invents a plausible-looking value for whatever it doesn't need. The trouble is, at the interface level, there's no way to tell an invented value from a real one.

Converting "about 2.2x faster" into how long the same website-building task actually takes to wait for: