Why harness engineering won't save AI coding

Common critiques blame flawed usage: weak prompts, poor workflows, or insufficient reviewer agents. This line of reasoning misses the fundamental issue.

While external harnesses can be modified indefinitely, the model's underlying weights remain fixed. These internal weights ultimately constrain harness effectiveness, as demonstrated by Claude Code.

Before Claude Code, CLI tools like Aider, Cline, and Codebuff offered similar toolsets and context management: reading, editing, searching files, and running shell commands. Yet they frequently failed on simple edits, forcing developers to intervene manually.

The 2024 SWE-Agent paper demonstrated that subtle tool design changes significantly impact performance—such as adding line numbers to file outputs or switching file editing from search-replace to line-range editing.

Comparison of tool designs. Left: No dedicated edit tool; the agent dumps the file and edits via shell. Center: Dedicated edit tool, but recovery from errors is difficult. Right: Edits pass syntax checks first; invalid edits are rejected with feedback to retry. Source: SWE-Agent paper (2024).

When Claude Code launched, its performance stood out. Rather than relying solely on harness tuning, Anthropic trained the model directly on the exact tool signatures it would use in production. Optimizing harnesses without model weight access cannot compete with co-designing models and harnesses together, a point echoed by OpenAI's team in late 2024.

Flaws in model training: No penalty for unmaintainable code

If harness engineering cannot bridge the gap, where does the issue lie? In RL training objectives, which contain zero penalties for unmaintainable code.

Grading harnesses check only two outcomes: Did it fix the bug, and did it break anything else?

The fundamental problem traces back to how models are trained under reinforcement learning.

The RL loop follows three steps: sample multiple execution trajectories for a prompt, grade each attempt using an automated test harness, and update model weights to favor successful paths. This loop repeats millions of times across weeks of training.

Harness: The execution environment surrounding a model—tool declarations, context formatting, retry logic, and evaluation loops. Changing the harness alone can dramatically shift performance.

Examining a benchmark task: "Code quality" is nowhere in the reward function

Consider SWE-bench Multilingual, an industry-standard benchmark derived from real bugs in repositories like Redis, jq, and Django. The grading harness checks only two binary criteria: whether failing target tests now pass (FAIL_TO_PASS), and whether previously passing tests remain unbroken (PASS_TO_PASS). Meeting both yields 1 point; otherwise 0.

In one fastlane (Ruby) benchmark issue, the program attempted to invoke `.empty?` on a `nil` object when optional parameters were omitted, causing a runtime crash.

'zip_command': undefined method 'empty?' for nil:NilClass

-      @include = params[:include]
-      @exclude = params[:exclude]
+      @include = params[:include] || []
+      @exclude = params[:exclude] || []

During evaluation, the model receives the error trace and codebase snapshot. It cannot see the reference solution or test suite patch. Upon submitting a patch, the evaluation pipeline strips any model edits to test files (preventing models from commenting out failing tests), applies the benchmark test patch, and executes the suite.

Complete evaluation flow: Bug report and pre-fix snapshot on the left, agent patch in the center, hidden grading step on the right (stripping test edits, applying official tests, running rspec). 1 point for passing target tests without regressions; 0 otherwise. Source: Dex Horthy.

Implementation quality has zero influence on the score.

After millions of training steps, models converge on antipatterns like wrapping code in try-catch blocks to suppress exceptions, or using double type-casts to bypass static type checking. Both pass the automated test harness despite degrading codebase integrity.

Swallows errors and returns null. Callers receive null with no context on failure.

Double type-casting to bypass type checking before attaching arbitrary properties. Source: Dex Horthy.

Why code quality cannot be easily rewarded: The bill arrives weeks later

Why not add code quality metrics to the reward function? The challenge lies in evaluation latency.

Architectural debt manifests as "shotgun surgery" (a concept formalized by Martin Fowler), where changing a single requirement forces scattershot edits across a dozen files while risking silent side effects.

By the time production outages occur, credit assignment cannot be backpropagated through time to the original design decision. The causal link between early shortcut edits and later outages is broken.

Broken credit assignment: Early shortcut decisions (pink) lead to brittle code (red), causing outages months later (right). The backward arrow is crossed out, showing backpropagation cannot assign credit across this gap. Source: Dex Horthy.

"It stands to reason that if the model were capable of reliably evaluating code quality, it would have generated clean code in the first place."

Dex Horthy, "Why Software Factories Fail"

Frontier research and industry efforts

Research teams are actively exploring alternate reward structures, a direction Dex considers promising:

While these benchmarks represent progress toward evaluating maintainability, Dex notes that automated reviewer agents and extra token compute raise performance floors without moving model ceilings. RL requires fast, reliable grading harnesses—which long-term code maintainability currently lacks.

Turning the lights back on: Restoring human code review

Because model capabilities remain bounded by RL reward limits, human code review must be restored. AI effort should shift upstream into pre-implementation planning, reducing downstream errors. In his team's experience, spending one hour on upfront planning reduces code review turnaround from six hours to 20 minutes.

Traditional software engineering reached a similar conclusion decades ago: because building and reviewing both demand substantial effort, upfront design aligns architectural expectations before code is written. This reduces rework and accelerates code reviews.

Software factory diagram with high-leverage points highlighted: 1 hour spent on upfront planning (bottom left) reduces review time from 6 hours to 20 minutes (right) while lowering rework rates. Source: Dex Horthy.

He structures upfront planning into four sequential stages:

Upfront four-stage AI planning before code generation

Treat current AI agents as fast, obedient junior engineers who lack global architectural perspective. Prompting them directly for complex features yields thousands of lines of passing yet unmaintainable code.

The highest-leverage point occurs before code generation. Every stage is drafted by AI and finalized by human engineers.

Stage 1: Product Review

A concise specification defining two elements: the specific user pain point (framed in user terms rather than technical jargon) and explicit success criteria (e.g., reduced flow completion time, onboarding milestone metrics, or lower ticket volume).

For UI-facing features, generate interactive HTML prototypes directly rather than relying on textual descriptions. A mock interface resolves ambiguities faster than lengthy discussions.

Product review document with an embedded UI prototype for creating a task (workflow selection on the left, step preview on the right). Source: Dex Horthy.

Minor fixes, copy edits, or one-off scripts bypass this step. Formal product review is reserved for high-rework-risk features. Additionally, the assigned PR reviewer reviews the spec before implementation begins.

Stage 2: System Architecture

This stage establishes integration boundaries—defining communication across services, API contracts, database schemas, and message queues prior to internal module design.

PUT /api/resources/:slug
  request:  { destination: string }
  response: { resource: Resource }

Dex notes that while diagrams like Mermaid are useful, text-based contracts and schema definitions often prevent model hallucinations more effectively.

Stage 3: Program Design

Relying solely on architecture leaves internal module structure to model discretion, often yielding suboptimal code organization.

Before code generation, developers define concrete program shapes: core types, function signatures, file tree diffs, and call stack trees. Dex emphasizes that program design is heavily underrated in AI coding workflows.

Their team utilizes three lightweight pseudo-code artifacts:

 entrypoint
   runCommand
+    handleCreateResource
+      ResourceClient.create(input)
+        POST /resources
+      renderResult
-    legacyCreateFlow

 src
 └── resource
+    ├── resource-client.ts      # NEW - wraps API contract calls
+    ├── resource-client.test.ts # NEW - covers request/response mapping
~    └── resource-route.ts       # MODIFIED - wires create action into UI

interface Item {
  id: ItemId
  parentId: ItemId | null
  // ...
}

interface Cursor {
  position: ItemId
  direction: 'up' | 'down'
  // ...
}

resolveTarget(items: Item[], cursor: Cursor) -> ItemId | null

These artifacts allow rapid iteration between developer and model. Unresolved structural details during planning inevitably trigger costly refactoring during code review.

Stage 4: Vertical Slicing

Rather than generating entire features end-to-end across database, API, and UI in a single pass, vertical slicing breaks work into thin, testable end-to-end increments.

AI models naturally favor horizontal layer-by-layer plans (database → service → API → UI).

Horizontal execution prevents intermediate end-to-end testing. Manual verification in the browser or terminal becomes impossible until the final layer is completed.

Vertical slicing establishes working endpoints with mock data first, builds UI views against mock responses, connects real service layers, and finishes with data migrations and edge-case handling.

Frontier models rarely slice work vertically without human intervention. Releasing 1–3 thin slices per iteration allows reviewing 100–200 lines at a time rather than debugging 2,000 lines at once.

Managing code review overhead in the AI era

Even with upfront planning, code review queues remain active. His key insight: "You don't have too many PRs; you have too many low-quality PRs."

Well-structured PRs that conform to team architecture conventions are straightforward to review.

Conversely, PRs requiring 20%+ rework impose significant cognitive load. Unguided AI-generated PRs often require near 50% rework.

Upfront planning reduces code review friction. To determine when full four-stage planning is warranted, he proposes a decision framework based on expected risk.

His team tailors planning depth by task scale: roughly 40% of small tasks use direct single-prompt agents, medium tasks combine product and architecture into a single plan, and full four-stage planning is reserved for major features.

He defines "Expected Pain" as:
Expected Pain = Probability of Rework × Severity of Rework

Evaluating three points along this curve: a 2-minute quick prompt carries a ~50% rework probability; 5 hours of detailed spec writing by a senior engineer drops rework probability to ~10%; spending 20 hours writing all code manually eliminates AI leverage entirely.

Expected Pain vs. Upfront Planning Investment. X-axis: time spent planning; Y-axis: expected pain. Conceptual estimates by Dex Horthy. Source: Dex Horthy.

He notes that minor UI tweaks carry low rework severity, keeping total expected pain minimal even if rework probability is high.

Accepting constraints to find true leverage

While fully autonomous software generation remains an attractive vision, current model limitations require structuring workflows around realistic capabilities.

Rather than chasing hypothetical 10x–100x efficiency claims at the expense of code quality, teams can achieve a sustainable 2x–3x throughput increase by adopting structured upfront planning.

He concludes with four key principles:

StrongDM responds to the article

Following Dex's comment regarding sporadic updates on StrongDM's public progress page, StrongDM AI Labs responded on Hacker News.

StrongDM engineer jaytaylor clarified on July 23 that updates occur monthly as milestones are met. They remain optimistic that software engineering challenges will succumb to automated factory pipelines as base models improve. Dex replied requesting a retrospective detailing successes and lessons learned.

Attributing code quality failures to weak prompts or missing reviewer agents overlooks fundamental model weight constraints.

Reinforcement learning optimizes models strictly according to evaluation criteria within the grading harness.

Incorporating maintainability into rewards remains difficult due to evaluation latency.

Frontier efforts like Abundant AI's SWE-Marathon (long-horizon partial scoring) and Cognition's Frontier Code (penalizing redundant tests) attempt to evaluate maintainability, though practical deployment remains early stage.

Even with upfront planning, code review queues remain active. His key insight: "You don't have too many PRs; you have too many low-quality PRs."