Lesson 1: Lock the “Worker” Down

Within a single evaluation or deployment cycle, keep three things fixed: the model, the agent's runtime, and the native tool interfaces. If any of these changes — even a generational upgrade — the whole environment needs re-acceptance from scratch.

Locking the black box makes failures attributable. When a capable agent fails, the cause is almost always one of five: missing institutional context, a capability that isn't actually usable, a missing feedback loop, insufficient authority, or no evidence at the exit. All five are properties of the environment; all five can be inspected and changed. Without a locked box, you stay stuck asking "is it the model, or my setup?"

Two numbers show how much a generational shift matters. When GPT-5.2 shipped, he was on vacation. He came back to find the team producing one to two extra PRs per engineer per day with zero additional environment investment. Later, a separate Symphony environment overhaul reported a 10x increase in PRs per engineer per week. The first came from a model upgrade, the second from environment changes. You have to track these separately — that's what locking the box enables.

Inner-loop latency is part of acceptance too. How well an agent reasons depends on how much feedback it gets before its trajectory runs off the rails. In the GPT-5.2 era, the Codex environment had no background shell, so the agent dutifully waited for blocking build scripts. When GPT-5.3 added a background shell, it stopped waiting. The team spent a week swapping custom build scripts for professional tools (trying Bazel, Turbo, and Nx along the way), stopping only when the full build fit in under a minute. That minute is now a hard cap — the moment it breaks, they stop feature work and fix the build graph.

Acceptance also includes subtraction. To let the agent "see" the interface, the team built a custom virtual display plus screen recording. Once the model could see the screen and click the mouse itself (computer use), they deleted the whole stack — without losing end-to-end verification.

Lesson 2: Deploy the Underwater Part of Your Private Process-Data

What's inside a general model's weights is just the tip of your organization's knowledge iceberg. What you actually need to get work done is below the waterline — private and constantly changing.

You need two things clear. One is the data ontology: what entities, metrics, and identifiers exist and how they relate. The other is the work ontology: what results, workflows, roles, tools, exceptions, evidence, and approval relationships exist.

An example helps. An internal data agent with only database access can't resolve the company's internal conflict over what "revenue" and "active user" mean. It also needs product lines, customer segments, teams, pilot customers, current plans, and how shared metric definitions are written. With those, it can give an answer that belongs to *this* company.

There's a pattern you can copy directly. The company Basis has two repositories: Arnold is the production monorepo, and Atlas is a second repository specifically for company context, separate from the production tree. Codex can combine personal notes, operational knowledge, and production code, while each source keeps its own update path. The implementation can be simple — his exact words: "just Codex plus a Git repo full of Markdown."

One more thing belongs to this layer: settled decisions should be promoted to the layer that can enforce them. A team and its security group picked an approved encryption implementation, and the decision stayed in Slack. Later, a new engineer was working with Codex, and Codex installed a different npm crypto package. The team pulled the decision back, wrote it as a guardrail rule in the repo, and re-ran that change.

Lesson 3: Give One Agent the Whole Job

Delegate the outcome. Give one main trajectory full responsibility: breakdown, execution, integration, evidence, and safe landing. Leave the steps to the agent. A person steps in at only three points: direction, judgement, and consequential authorization.

He often gives near-zero-constraint prompts, calling them "lazy prompts" — lazy about method, leaving it to the agent to decide how. Sparse delegation is also a test: it tests whether your environment makes the real requirements findable by the agent itself. When the agent misreads you, what gets exposed is exactly which part of your environment isn't built yet. The accompanying loop is "propose, observe, improve, rerun": review the trajectory, fix the environment, discard that result, run it again.

Two counterexamples, more useful than the positive one

A single request usually describes just the functional outcome. Compatibility, maintainability, accessibility, rollback, syncing artifacts, and future maintenance needs are all implicit. These implicit requirements have a name: "the question behind the question." The agent needs to find them itself — by searching the repo, looking at current behavior, checking history, and understanding how this result relates to the user. Only when it can't dig them up, and the missing answer would change the product intent, create unanticipated risk, or require permissions it doesn't have, should it ask a person.

Lesson 4: A Thousand-Page Manual, Replaced by an Index

The knowledge base should be large and searchable; the working set should be small. The root guide should be a map.

This lesson has an origin story. Their original monolithic agent manual got too long to be usable. They replaced it with a short root index, backed by indexed design docs, product specs, implementation plans, and references — making progressive disclosure a feature the repo maintains itself, and using link checks and freshness checks to keep those routes trustworthy.

A bigger context window solves only capacity pressure. Attention, retrieval quality, conflicting instructions, and "which fact matters right now" remain scarce. Deterministically stuffing the context will inevitably fail across multiple compressions. The agent must have the ability to find its way back; counting on the first injection still being there is unreliable.

How to validate a knowledge base

Start a brand-new session and see if the agent actually retrieves it and actually changes its behavior. A document existing only proves it was written. One step further is an ablation test: remove this piece of context and run it again. If the result holds without it, delete it.

This repository is itself a sample of this lesson

Four structural facts you can verify for yourself:

The unresolved decision determines the route. The target system's local evidence might resolve it without any of these sections. If context is genuinely needed, start with one section. Add a second only if a different independent decision is still pending. AGENTS.md, repository root

Lesson 5: Every Word a Tool Spits Out Is Context for the Next Step

For a capability to be truly usable, it has to complete a six-step loop: be discoverable, be recognizable for this task, be callable correctly, have return values you can interpret, be fixable when it errors, and finally be confirmable in the real system. If any link breaks, the capability is ornamental.

If you find an error at the "Read output" stage, loop back to fixing it, then call again.

The choice between MCP and plain command-line tools comes down to two trade-offs. MCP servers instantly broadcast tool names and input schemas, but the model still has to learn each new call and return shape on the fly. Ordinary POSIX commands benefit from the model's strong prior, but they're invisible until the agent knows they're installed. There's a real migration story: they initially used MCP to let Codex connect to their Electron app via the Chrome DevTools protocol. A colleague later swapped it for a local TypeScript daemon plus a small CLI, because they really only needed two or three operations. The workflow didn't break after the switch.

Seven behaviors of a good tool

A trap everyone steps in

There's a failure mode that's easy to recognize: the agent will preemptively pipe an expensive command, like cargo test, into head or tail to read less output.

The fix isn't in the prompt; it's at the tool boundary: a wrapper owned by the repository itself, which runs the whole task, preserves the true exit status and full output, returns only the semantic failure summary the next decision needs — while keeping the low-level evidence queryable, without re-running.

Dig down until it works

Handing over the whole job exposes capability gaps layer by layer, from the outside in. The initial goal was to let Codex read Slack, identify incoming production incidents, so people wouldn't have to paste messages into the conversation one by one.

This capability search recurses six to eight levels deep. Once you hit a usable primitive at the bottom, you climb back up to the original triage goal. Every time you dig this way, the set of tasks all future agents can do expands a little more. Conversely, using a one-off manual workaround leaves the exact same hole for the next time.

Lesson 6: Code Is Prompt

Every file an agent reads influences what it predicts next. His quote: the code in the repo is the prompt.

Models have dozens of plausible approaches in mind. The repo's job is to prune — using recurring structure, to single out the one approach the team has chosen. Consistency directly reduces context requirements: with one observability setup, what an agent learns in one file transfers across the repo; with six setups, it has to re-learn at every call site which one to use here.

The reverse is also true: examples propagate defects just as fast as good judgement. A half-finished migration produces two conflicting sets of prompt material. A poorly written helper function spreads to every package needing similar behavior.

One concept, one home

Before creating a parser, enum type, version constant, state representation, fixture constructor, policy check, or command — find the existing home for that concept, and extend that interface if needed. These are signals that something has no clear home:

Every duplicated home creates a "locally plausible" direction for the next edit. An agent can faithfully follow every precedent around it while making the entire repo less coherent.

Lesson 7: Can-Do and Can-Approve Are Two Contracts

Capability says how to cause an effect. Authority says which identity, on which resource, in which environment, may cause which effect, for how long, with whose approval, and how to roll back.

Splitting these two contracts shifts the security question from "require a random system to be error-free" to "how big is the consequence, and can it be recovered." Credential handling should live outside the trajectory: a proxy process, at the action boundary, pulls narrowly-scoped keys from secure storage, making gh feel like an ordinary command — while the keys never enter the model's context, and never touch the repo. Two more layers of protection: a read-only identity makes checks useful but unchangeable, and endpoint-bound access lets you revoke one agent-facing channel without rotating the underlying credentials — even revoking the authority of a running trajectory.

Phase consequential operations, grant only what each phase needs

Loki is his robot vacuum at home, running open-source Valeto firmware. The only remote access path is a custom Tailscale daemon. Upgrading it from v1.90.8 to v1.96.4 was tricky: upgrading it might destroy the very path you'd use to rescue a failed deployment.

Instructions should be read through this authority contract. "Merge" means completing the repository's protected merge process; it doesn't imply permission to bypass it. "Prepare a deployment" grants scope up to preparation; the production switch requires separate approval. When an irreversible goal or scope remains ambiguous after checking local policy, the agent should ask before crossing that boundary.

Settled authorization rules should land in identity policies, typed domain operations, scoped tools, environment boundaries, test and merge gates. Where the consequence warrants it, making the wrong action impossible is better than writing a rule that depends on someone remembering it. Mechanisms don't need remembering.

Lesson 8: A Green Check Proves Only Its Own Line

Verification must happen at the boundary where the user, operator, or downstream system actually depends on the result. Unit tests, type checks, linters, and builds establish internal properties. They don't prove the external thing.

The most useful artifact for this lesson is a mapping table: what you claim, and where the evidence must be taken.

This table holds for non-engineering work, too. When the intermediate code stays hidden inside the agent's execution and the user never sees it, the proof package must speak the user's language: a spreadsheet formula, a rendered slide, a sourced conclusion, a completed state transition. A hidden script's test passing only establishes internal properties.

Also, hand the real system to the agent itself. It should be able to start the app, drive the browser, read logs, query metrics and traces, check persistent side effects, compare generated corpora, and watch CI/CD. A person shouldn't relay evidence the agent could see directly — relaying adds latency and strips the state needed for diagnosis. Use both semantic and rendered views: DOM snapshots and structured state are good for roles, values, and relationships. Screenshots, video, or direct screen interaction are needed to expose layout, clipping, focus, and interaction defects.

Be explicit about what the evidence doesn't prove

In the security advisory case above, the report clearly stated: triggering the data corruption required a custom hash implementation plus recovering after a crash, and the reproducer did not prove memory unsafety on the default path. The instruction was that every reported issue had to prove impact or exploitability; speculative output that merely looked like a security finding didn't count.

The burden of proof scales with the consequence. A typo fix, a parser migration, a remote upgrade, a production release — these need wildly different levels of evidence and authority.

Lesson 9: What You’ve Corrected Once, You Shouldn’t Correct Twice

Recurring corrections should be turned into the environment that shapes the next run. This is what makes everything else compound over time.

Let's start with a precise counterexample. Someone gave an agent a specific Go interface fix: instead of returning just "did it succeed," return a concrete error value that explains *why* it failed. The agent changed only the named function. But the instruction implicitly carried a general principle about interface design. The fix didn't search for other instances governed by that same principle.

When you hit a recurring correction, treat it as a systemic problem and go through five steps.

Which layer to fix depends on how mature the lesson is and how big the consequence.

A full chain of how judgment becomes an asset

The starting point was a PR to the public openai/codex repository: upgrade pnpm to fix several vulnerabilities and disable post-install scripts. Working backwards from that patch, four principles can be recovered: package managers are high-risk boundaries, dependencies should stay updated, install processes shouldn't execute arbitrary code in CI, and exceptions need human review.

The second promotion is the notable one. After the scanner went live, Slack feedback showed it was better at catching individual instances than at capturing the general reference framework in the security engineer's head. The team added the missing guardrails and investigation signals to SECURITY.md, which then fed both the daily scanner and the new dependency-depth and license scanners. That's how judgment gets accumulated, round after round.

Lesson 10: Preserve Coherence, Carry the Forever Risk

When implementation becomes cheap, what gets scarce is future coherence and long-term ownership. Every accepted change is still adding constraints and maintenance obligations to a shared system.

This lesson contains the most honest passage in the entire piece. That five-month, million-line internal product — the internal launch and adoption strategy proved viable in the startup phase. But how architectural coherence evolves on a "multi-year" timescale is an acknowledged, unsolved problem in the original text.

Another honest place is where he uses his own project to push back on the impression that "agents can chew through hard work in one go." The Artichoke state-machine refactor started as a giant change across 114 files that didn't even compile. It was broken up into 50 preparatory PRs merged over time, with 200 other PRs landing in the repo during the same period. Then it took several more major integration attempts and runtime failures before it finally merged.

The size of the change and the runtime duration say almost nothing about whether a safe route has been charted. The difference is whether the route itself is certain. He gave this missing capability a name: "the fear of fifty more PRs" — predicting how much you'll regret a future change. He explicitly lists this as still an unsolved capability problem.

What has to survive a rewrite

The implementation itself is a disposable render. These six things are what you keep:

Symphony is built this way: it publishes a behavioral spec, and other teams can produce their own independent implementations. This pattern is called a "ghost library" — sharing a contract, not a code artifact.

Lesson 11: Once a Direction Is Settled, Run It as a Continuous Loop

There's a whole class of work where the direction is already decided, it just never fits into the schedule: localization, adding instrumentation, keeping dependencies updated, removing obsolete compatibility paths, fixing architectural drift, refreshing generated data and docs. Once the desired outcome, the evidence method, and the authority boundary are all clear, this work can keep pushing itself forward.

To see if a loop can stand up, check if it can answer five questions.

"Task" and "goal" are different here. "Fix this flaky test" ends with a patch. "Keep CI p95 latency under N minutes" is a goal — it continually observes the system, generates and tests interventions, requests human authority when needed, and has an explicit retirement condition.

There's also a claim about incident response: a code red shouldn't end with metrics turning green. It should leave behind a maintenance loop. During the response, people learn which invariants matter, which signals predict drift, and which interventions work. These should be recorded, along with the goal, sensors, authorized actions, escalation path, and the owner of the loop. Because agent execution is cheap enough, some newly discovered invariants can be maintained without a dedicated team.

The environment itself also grows weeds. Docs age, code grows conflicting styles, review comments expose missing guardrails, a once-useful tool becomes redundant after a model change. At OpenAI, a set of persistent background tasks replaced the old weekly manual cleanup: scanning for deviations from established principles, updating quality ratings, opening focused refactor PRs, cleaning stale docs. Small patches should land before a bad style becomes future prompt material.

This lesson has a clear role for people: The loop maintains a chosen policy's relationship to a changing world. "Which world is worth building" remains a human job.

Lesson 12: Optimize for Measured Effectiveness

Tokens, lines of code, number of agents, number of checks — these are inputs. What you want to optimize is accepted outcomes per unit of scarce human attention.

Token leaderboards are one vanity metric: the signal they reward goes up even when value stays flat. The "a billion tokens per engineer per day" number later got a gate added: the work has to merge to main and demonstrably improve the user experience to count.

Three concepts need separating: ReachHow much of the job an agent can actually touch. Repo, local app, tests, browser, logs, deployments, evidence — all expand reach, activity (tokens and busy time), and Effective utilizationThe portion of activity that actually lands an accepted outcome or produces reusable evidence. Preventable discards, rework, and fixes all cut into it.

You must track four clocks separately

Two examples side-by-side make clear why they must be separated. That 60-hour refactor had a long actual duration — three to four days — but a tiny synchronous human interface: just the initial prompt and two follow-ups. It consumed ~330–350 million tokens and $2,800, produced one PR, and he estimates it would have taken him three weeks to do by hand. Contrast that with a 2023 pairing session with ChatGPT: code copied back and forth between editor and browser, test results relayed by a human back into the chat box, a few hundred prompts per session. The actual duration wasn't long, but a person's attention was drained within an hour.

Exploration, rework, and waste are three different things

Discarding can be productive. In a high-uncertainty refactor where the final interface shape wasn't settled, he'd have the agent produce an exploratory change up to 50,000 lines, push it as a PR for human inspection, and then discard it. That attempt exposed where an agent would likely fail and what interface seams the eventual switch needed. He turned that failure map into about fifteen preparatory and enabling PRs, and only then came back for the real switch.

How to Apply These Twelve Lessons to Your Own Project

The repo's playbooks directory has a ready-to-follow process for a bounded, replayable, real task.

The "fresh" in step five has a specific meaning: use a new session and an isolated, equivalent starting state, so the re-run doesn't inherit hidden help from the original implementation conversation. Also verify separately that this intervention was actually retrieved, actually called, and actually had anything to do with this run. A successful result is zero evidence for an instruction or tool that was never used.

Three templates you can copy directly into your own project.

Target system & version:
Relevant external state:
Locked model & agent config:
Representative task:
Acceptable outcome:
Evidence that proves that outcome:
Authority scope & approval boundaries:
Budget & stopping conditions:
Suspected missing piece:

If we add [intervention] at [the earliest place that controls it],
then the locked agent on [representative task]
will show [observable behavior change], because [mechanism].

Evidence that could support this hypothesis:
Evidence that could weaken it:
Expected maintenance cost & ownership:

Task & acceptable outcome:
Target version & external state:
Locked agent & authority scope:

Baseline evidence:
The earliest broken handoff, and who owns it:
Change & expected mechanism:
What verification was done:
Evidence from the fresh re-run:

Outcome comparison:
Evidence comparison:
Human-relay & latency comparison:
Risk & maintenance cost comparison:
If you did a "remove it and run again," the result:

Decision: Keep | Modify | Remove
Long-term ownership & recorded lessons:
Follow-up actions or retirement conditions:
Known limitations:

When your comparison doesn't count

There's also a separate invalidation checklist. If any one of these is true, the evaluation can't support the conclusion it's aiming for.

The twelve lessons split into three layers: three set the boundaries to make an accountable environment, six are the operational levers you turn daily, and three decide whether this whole system gets more valuable over time. Locking the boundaries is the starting point. When the model generation or agent config changes, the entire environment needs to be re-accepted. Otherwise, when a task fails, you can't tell whether it's the model or your setup.

The code in the repo is the prompt you feed your agent. A pattern that appears over and over gets copied. Two patterns coexisting means both get copied. Corrections work the same way: a one-off, point-specific fix that only changes the named thing means the same class of problem will recur elsewhere.

The full article also has three fill-in templates you can copy directly, plus a nine-item checklist for "when your comparison doesn't count."

These numbers are all from Ryan Lopopolo’s own projects, not independently audited.