Why one-shot Verilog from an AI falls far short

Verilog is the hardware code used to describe chip logic. RTL design is a sharp touchstone: whether a module is correct depends on the exact behavior at every clock tick, the reset convention, how the interfaces handshake, the bit-width of every signal, and the feedback from an actual simulator run. So "Verilog that looks plausible" is nowhere near enough.

A model has to infer the datapath bit widths on its own, how the state machine transitions, the reset convention, the ready-valid handshake protocol, how memory is read and written, and all the edge cases that natural language often leaves unstated. Put simply, a syntactically correct module is just the starting point. Useful automation has to connect "writing it" all the way through to "compile, simulate, inspect waveforms and failure traces, then fix" — running the loop, finding what's wrong, going back to fix it, and running again.

HORIZON's core idea is to lock this entire chain inside a loop that runs itself.

How a spec becomes a self-running project pack

The only input the user has to provide is a structured Markdown spec — not code. It states the high-level intent, repository context, expected artifacts, evaluation criteria, and domain knowledge. Domain knowledge is especially useful, since it can make explicit the invariants, tooling conventions, and failure modes that simply can't be inferred from the files alone.

A "bootstrap agent" compiles this spec into a "project pack" — the control plane for the entire system. The spec has four components, which get compiled into a five-element runtime configuration that drives the unsupervised loop.

# HORIZON Harness: fifo_sync

## Goal / objective
Implement a synchronous FIFO. Depth 16, 8-bit data.

## Domain-knowledge directions
- Reset is synchronous, active-high.
- full and empty must never assert together.
- Follow ready-valid handshake conventions.

## Evaluator specification
- Compile with the suite's native flow.
- Run the provided simulation testbench.
- Extract functional coverage where available.

## Acceptance predicate
- Simulation passes with zero mismatches.

How the loop runs: the Git repo is both the operating console and the experience bank

Once the user hands off the initial spec, the loop is completely unsupervised: bootstrapping, generation, evaluation, acceptance, logging, next iteration — it all runs automatically. A single "campaign" can run for many rounds back to back without any human stepping in. Each round, the agent reads the current state, plans a goal, edits the workspace, invokes tools, interprets failures, and then either fixes the code or commits. The acceptance predicate decides exactly one thing: whether the new version gets committed, or the failed attempt just gets logged.

Git here is the substrate of the loop, not an afterthought for record-keeping. Four ordinary git operations each play a distinct role:

git diff --cached                        # inspect the staged candidate change
git commit -m "iter 7: fix full/empty overlap"
git notes add -m "pass=1 mismatches=0"   # attach evaluation evidence
git log --oneline                        # replay the search trajectory

It's the code that evolves — the model never moves

The term "self-evolving" can easily make it sound like HORIZON is secretly training the model. It isn't. What evolves is the design artifact inside the Git repo, not the model's weights. The backbone model, GPT-5.3, stays fixed throughout the entire campaign — it's neither trained nor does it update any reinforcement-learning policy.

The paper does borrow the term "semi-Markov decision process (semi-MDP)," but really, it's just a precise, replayable name for the objects being recorded: a "state" is a versioned snapshot of the repo (a bookkeeping checkpoint), and an "option" is an entire episode between two checkpoints. It's not a behavioral assumption, and it's not an optimization assumption.

Cost is kept down through session reuse: HORIZON maintains a persistent model session across iterations, so the spec, project pack, stable source files, and accumulated debugging context are all pulled from the provider's prompt cache instead of being resent every round. So the tokens that actually get newly billed are mostly the current diff, the latest evaluation output, and the agent's reply. That's also why, as we'll see below, the vast majority of cumulative tokens are cached input.

Whose shoulders HORIZON stands on

HORIZON continues a lineage of "repository-scale self-evolution." This lineage shares one common principle: a candidate change is only accepted when executable evidence supports it. The difference lies in "what evolves" — earlier systems evolved software that engineers "run"; HORIZON evolves the hardware artifacts that engineers "create."

The whole benchmark suite hits 100%, but the roads to get there differ wildly

Running as a single unsupervised agent loop, HORIZON drives every benchmark suite to a 100% pass rate. The one item that remains "unsolved" is a single ChipBench problem, which the authors trace to a mismatch between the spec and the original benchmark's own defect, rather than a failure on the agent's part — so counting it as resolved still yields 100%. So what varies across suites isn't the finish line, but the road it takes to get there.

Here, iter 0 isn't an independent single-shot success rate. It reflects the state of the repo right after the agent's first evolution step, using the exact same prompting strategy throughout. The agent could easily choose to deliberately push most of its exploration, debugging, and fixing to later iterations rather than trying to maximize success on the very first attempt. This aggregate figure is also pulled up by Verilog-Eval-v2 (whose iter 0 is 86.2% on its own) — looking at the CVDP subset alone, iter 0 is only 23.9%.

Worth noting: these 9 CVDP verification categories (testbench generation, checker generation, assertion generation) were previously exactly the tasks where AI struggled the most. The CVDP paper itself points out that state-of-the-art single-shot models particularly struggle on these verification tasks — and now the same loop converges them all to 100%.

Pick a benchmark to see its convergence curve from the first round to 100%:

Lining up the first-round pass rates for all 12 categories side by side makes the gap unmistakable: from 86.2% for Verilog-Eval down to 0.0% for CID 007. And every single one of them ultimately converges to 100%.

Bar = first-round (iter 0) pass rate; every category eventually converges to 100%.

Once correctness saturates, tokens become the real competition

Once correctness saturates, token consumption becomes a much more informative signal — it exposes what's actually hard versus easy. The authors measure this as "cumulative tokens up to the optimal iteration." The three traditional suites together use only 6.0M, while the 9 CVDP categories use 203.9M — 97.1% of the total. The full run comes to roughly 209.9M.

Spreading the 209.9M across categories, code completion (CID 002), spec-to-RTL (CID 003), and stimulus generation (CID 012) together account for more than half, while the traditional benchmarks are squeezed into a thin sliver on the far left:

What's striking is CID 013: its first-round pass rate is the lowest overall (3.8%), yet it converges on just 14.2M tokens, confirming that smooth, plateau-free trajectory. What actually drives up cost is the "long tail of a handful of stubborn designs" like CID 002.

One more key point: roughly 91% of tokens are cached input, which substantially lowers API cost. So the authors explicitly state that "token efficiency" — not "final pass rate" — is the metric that most needs improvement. Even though HORIZON clears every single suite, the last few percentage points on the hardest categories eat up a disproportionate share of the budget.

Coverage is measured as a byproduct; the red lines are the authors' own

The verification-generation categories are worth a closer look, since they reveal what HORIZON is and isn't optimizing for. HORIZON's acceptance gate is CVDP's "pass condition," not a "coverage target." The loop is driven to make the benchmark spec pass; once a given design passes, the gate is satisfied and the loop stops trying to further optimize it. So coverage here is a secondary, observational signal.

The most direct evidence: CID 012 reaches a 100% pass rate at round 32, but at that point its average per-design coverage is only 97.9%, not 100%. This gap is a direct consequence of the acceptance gate — each design stops the moment it passes the spec, so coverage reflects only "testing sufficient to pass," not "the maximum achievable."

Looking design by design, the loop lifts a "low-coverage tail" — several designs that start at 20% to 40% coverage get pulled up toward full coverage, rather than the loop only nudging the ones that were already high. CID 014 is the counterexample: it starts near 98% right away and saturates immediately. The authors stress that they never tried to force coverage to 100% — here, coverage is a diagnostic for "the generated tests have real substance," not a claim of "exhaustive verification."

Section 5 of the paper is its most candid part, where the authors proactively draw two red lines.

Red line one: over-fitting to the reward／reward hacking

In the current setup, the agent can see detailed evaluation output from every round — simulator messages, evaluation logs, failure traces. This mirrors real debugging workflows, where engineers naturally check logs and counterexamples. But full access to these signals also opens up a failure mode: the agent could tailor its generated RTL to exactly match the observed failures, deterministic tests, or evaluator quirks, rather than robustly implementing the design's actual intent. So a "pass" result might mean "the spec was satisfied under the exposed traces," rather than "the spec was satisfied under all reasonable tests."

Based on this, the authors recommend that future agentic hardware-design benchmarks adopt a two-tier protocol: expose useful diagnostic feedback during the fix cycle, but reserve hidden randomized tests, independent reference models, formal equivalence checks, property suites, or held-out simulator configurations for final scoring. This lines up with the "structural test holdout" approach used by the software-engineering benchmark SWE-bench.

Red line two: feedback turnaround time

The pass／fail benchmarks in this paper are relatively forgiving, since most evaluations run fast enough to support iterative fixing. But in broader chip design, reward evaluation can be far slower — PPA (performance／power／area) optimization or EDA tool-evolution rewards may require synthesis, place-and-route, timing analysis, power estimation, or large regression suites. The authors note that SATLUTION has already shown the cost of this: even with roughly 800 nodes running in parallel, evaluating the full SAT competition benchmark takes about two hours to turn around, while PPA optimization for RTL could take turnaround times of days or even weeks. Long-latency rewards fundamentally change the problem — a naive "edit, evaluate, fix" loop becomes too slow, and the agent has to reason under delayed, sparse, expensive feedback. Solving "long-turnaround rewards" is therefore a key research challenge for agentic chip design.

HORIZON's answer: the user just writes one Markdown spec (the goal, the domain knowledge, how to judge a pass), the system automatically compiles it into a "project pack," and hands it to an unsupervised (nobody monitoring it) loop that repeatedly edits code and runs simulations, committing only once verification passes. Across four public chip-design exams — 12 categories in total — this one loop pushed every single one to a 100% pass rate.

But with the same loop, some problems clear in two or three tries while others grind on much longer — where exactly does that gap come from? First, see how it actually runs.

The trick isn't a smarter model — it's treating the Git repository (the tool programmers use to store code and keep a history of every step) as the brain. Every time a new version is edited, a simulation runs: if it passes, git commit stores it as a green checkpoint and the main line moves forward; if it fails, git log records it as a cautionary example. This history in the repo becomes the AI's own experience bank.

All 12 exam categories hit 100% — something no unsupervised AI had ever done before. But behind that "100%" the problems vary wildly in difficulty; the clearest yardstick is "how many versions did it take to pass."

All figures above come from NVIDIA's own paper testing, with no independent third-party reproduction yet; the authors also caution that the AI can see detailed evaluation feedback each round, which means it might be "gaming the exam" rather than truly getting the chip design right.