A Verification Workflow: AI Coding Agents Drive the Browser, Verify UI Results, and Deliver Evidence

This verification workflow enables AI coding agents to launch a browser, click, type, save, and reload along real user paths after writing frontend code—checking functionality and UI rendering while leaving behind reviewable evidence.

Developer Timothy Ogbemudia initially codified these requirements into project rules and a reusable Skill, later developing the open-source tool katalepsis to execute checks and compile reports. The verification workflow can also be used independently.

Anyone using Claude Code, Codex, or Cursor to modify web applications has likely encountered this scenario: the agent reports, "Implemented, all tests passing," yet opening the page reveals stale data after filtering, settings reverting to defaults upon reloading, or a modal save button pushed off-screen on mobile viewports.

That leaves you repeating the user flow yourself, taking screenshots, pasting error logs, waiting for edits, and opening the browser once more to verify. While code generation has accelerated, verification still relies on humans acting as messengers.

Ogbemudia set out to close this feedback loop: after modifying code, the agent runs existing checks, launches the application, exercises the newly modified features, fixes any discovered issues, and tests them once more. Existing test suites inspect their predefined scenarios, while browser verification investigates the runtime consequences of current changes. Working together, they catch the common failure mode where "all tests pass, yet the interface is broken."

Walking Through a Verification Pass: Modifying Settings, Saving, and Reloading

An illustrative pedagogical scenario walks through the workflow below: a settings page adds saving capabilities for a "Display Name" and a "Weekly Digest" toggle. The routes, fields, and subsequent report are teaching aids rather than an executed project.

1. Read the Request and Changes to Define Verification Scope

The agent first runs git status --short to record pre-existing modifications and untracked files, then examines the requirements alongside actual code diffs—including commits relevant to the task, uncommitted edits, and newly added files—without mistaking a single git diff HEAD for the complete scope of changes.

Assuming the modifications involve a settings page and its persistence API, the verification plan must articulate specific target outcomes:

• Open /settings/profile and record the initial state of the name input and the toggle switch.
• Modify them to new values distinct from their initial state, save, and inspect the immediate outcome.
• Reload the page, wait for data loading to settle, and inspect the current runtime values of the name input and toggle switch.
• Verify that save-state disabling resets properly and that meaningful error notices appear on failure; incorporate empty states or adjacent routes if affected by the changes.
• Inspect layouts across relevant desktop and narrow viewports, checking save network requests alongside any newly introduced console errors.

Before opening the browser, run the project's existing relevant tests, as well as required build or type checks, recording their results. Browser checks exist to answer the questions these automated suites leave open.

2. Start the App Using Existing Conventions and Track Created Resources

First check the dev script in package.json, then consult the local development instructions in the README; if containers are required, use the project's existing Docker configuration. Commands, port numbers, and dependencies should strictly follow project conventions.

Preserve logs during startup, recording the PIDs of spawned processes, running containers, and temporary directories. Wait for server readiness within an explicit timeout limit. If startup fails, inspect the logs and report missing prerequisites rather than declaring the application ready simply because a process was created.

Verification targets the authentic application prepared for delivery. Never disable authentication, modify .env, mock services arbitrarily, or inject temporary bypass routes merely to simplify login. If documented test credentials or seed data fail to permit login, report the step as blocked and state why.

3. Modify, Save, Reload, and Verify Final State

Once browser access is established—detailed later—the agent navigates to the target route. In this example, assume the initial display name is "Original Name" and the weekly digest toggle is disabled:

1. Confirm initial values, then update the name to "小互" (XiaoHu) and toggle the weekly digest to active.
2. Click save, checking for success notifications, button state changes, and whether corresponding network requests succeeded.
3. Reload the page and wait for settings to finish re-fetching.
4. Read the current value of the input element to confirm it remains "小互" (XiaoHu); inspect the toggle state to confirm it remains active.

This requires inspecting the live runtime control state rather than merely reading the static HTML value attribute from initial markup. The DOM represents the browser's live element hierarchy; through it or accessibility snapshots, agents can locate controls and inspect their state. In Playwright, the name input can be verified with toHaveValue("小互"), and an authentic toggle or checkbox with toBeChecked(). When using generic browser tools, corresponding runtime properties must likewise be read from the reloaded page.

This sequence verifies the actual result of modifying values, saving, and reloading. "The button was clicked" or "a toast appeared" cannot substitute for verifying persistence. Playwright web assertions auto-retry conditions until a timeout, making them ideal for waiting on asynchronous UI updates; assertions must nevertheless map to real user expectations.

4. Inspect State for Functionality, Screenshots for Layout, and Responses for APIs

Different forms of evidence answer different questions:

• Functional state: Inspect the DOM or accessibility snapshot to verify names, selection states, button disabled flags, and modal visibility.
• Visual rendering: Capture screenshots across target viewports, such as 375×812 and 1440×900, to check for obscured buttons, clipped dialogs, or overlapping text. These represent browser viewport simulations, not testing on physical mobile hardware.
• APIs and new errors: When saving depends on backend endpoints, inspect response payloads and status codes alongside console errors introduced during the interaction. A page displaying a success banner while its persistence endpoint returns an HTTP 500 must still be reported as a failure.

Screenshots demonstrate visual clipping but cannot independently prove data persisted after reload; similarly, a modal existing in the DOM does not guarantee it is interactable on screen. Scope must center on the current change: pre-existing warnings or unrelated network requests should not balloon into separate bug-fixing tasks, nor does a pure frontend toggle require an exhaustive network audit.

A similar discipline applies to filtering and resets. In his freeCodeCamp guide, Ogbemudia details a filter verification plan: record the initial row count, filter by Active, stack an Owner filter, clear them to verify the original dataset restores, then reapply non-default filters to verify URL query parameters and filtered rows survive a page reload. Clicking "Reset" when no filters were applied in the first place and claiming "reset works" proves nothing.

5. Fix Defects and Re-run the Verification Flow

If reloading reverts the input to its old display name, retain the captured network requests, control states, and error logs, then return to the codebase to investigate the cause. After implementing a fix, rerun affected test suites and repeat the entire sequence: "modify → save → reload → inspect live state." Never declare "it should work now" immediately after editing code.

Missing required test credentials, migrations, or an available preview environment may leave verification blocked. However, if an executed product flow fails because a real dependency API returns an error, retain the actual failure and its cause rather than relabeling it as blocked merely because the issue lies outside the immediate codebase.

6. Preserve Deliverable Evidence and Clean Up Only Agent-Created Resources

Whether a run succeeds, fails, or blocks midway, close the browser instances created for this run, terminate agent-spawned services and containers, and remove ephemeral artifacts that do not belong in hand-off deliverables. Clean up strictly according to the resource provenance recorded at startup, avoiding broad sweep commands like pkill node or git clean -fd that risk collateral damage to other running processes.

Inspect git status --short once more: pre-existing modifications and untracked files must remain untouched, the current bug fix should be explainable, and transient QA artifacts must never be bundled into code commits. Screenshots and execution traces destined for the report should be saved in dedicated evidence directories with their paths documented. Because bug fixes naturally alter source code, the working directory does not need to be completely identical to its starting state. The verification workflow itself does not authorize making additional commits or pushing to remote repositories.

7. Deliver What Was Done and What Was Observed

The final hand-off must detail visited routes, executed actions, expected versus actual outcomes, evidence locations, related test results, applied fixes, and any blocked or untested items. A human developer should be able to review the results directly without having to ask, "What exactly did you test?"

A Reviewable Delivery Report

The following markdown is an illustrative pedagogical sample corresponding to the scenario above, not an actual test output generated during this session:

### Settings Page Verification Report (Illustrative Example)

- Environment: Local development application; route `/settings/profile`.
- Initial State: Display Name "Original Name"; weekly digest disabled.
- Pre-checks: Existing project tests for the settings page executed and passing.

Actions & Results:
1. Update Display Name to "小互" (XiaoHu), enable weekly digest, and click Save.
   Expected: Success notification appears; save button returns to enabled state.
   Actual: Matches expectation; save network request succeeded.
2. Reload page and wait for settings to finish re-fetching.
   Expected: Current Display Name remains "小互" (XiaoHu); weekly digest remains enabled.
   Actual: Read live input value and toggle state; matches expectation.
3. Inspect desktop and narrow viewports.
   Expected: Buttons fully visible; labels do not overlap.
   Actual: Matches expectation; no new related console errors introduced during this run.

Findings & Re-verification:
- Initial run revealed the save button stayed permanently disabled after saving.
- Fixed submit state reset logic, reran affected tests, and repeated the full flow (modify, save, reload); all checks passed.

Evidence (Sample Paths):
- evidence/settings-desktop.png: Desktop layout screenshot.
- evidence/settings-mobile.png: Narrow viewport layout screenshot.
- evidence/settings-actions.log: Log of actions, field states, network requests, and errors.

Untested / Blocked: Physical mobile devices not tested; no items blocked during this run.
Cleanup: Terminated agent-spawned browser, dev servers, and temporary files; preserved above evidence.
Workspace: Pre-existing modifications and untracked files preserved; bug fix documented; no transient QA artifacts staged for commit.

Embedding the Process in the Project: Concise Rules, On-Demand Skills, and Independent Reviews

Relying on developers to paste a long prompt every time leads to omitted steps. Ogbemudia's approach encodes completion standards into persistent project rules while placing actionable procedures into an on-demand skill.

Persistent Rules Define "When It's Done"; Skills Define "How to Check"

In clients supporting corresponding conventions, files like AGENTS.md (used by Codex) or CLAUDE.md (used by Claude Code) load working instructions based on client and directory scope. Inlining an exhaustive browser manual into these files adds unnecessary context overhead to tasks that never touch the browser.

A Skill is a directory containing a SKILL.md file that begins with metadata such as name and description. Compatible clients typically display a concise summary first, reading full operational steps only when matched or explicitly invoked. This enables a clean separation of concerns:

• Project Rules: State that user-visible behavior must be verified in the running application, referencing which Skill to follow.
• browser-review Skill: Detail the complete operational sequence—from reading diffs to launching, interacting, gathering evidence, fixing regressions, and cleaning up.

These files serve as agent instructions, with actual loading behavior governed by client support. The original author worked with both Codex and Claude Code, sharing rules via Claude's @ import syntax; this is one configuration pattern rather than a universal cross-client behavior.

When AGENTS.md is supported, the custom rule block below can be added; for Claude Code, place it in CLAUDE.md or import it via its supported syntax:

## UI Verification Standards

Before declaring user-visible changes complete, verify affected flows in the running application.
Follow the procedure in `.agents/skills/browser-review/SKILL.md`.
Report build, type check, and existing test results alongside live browser verification findings.
Explain the reason for any blocked steps; list omitted checks as untested rather than claiming a pass.
Clean up only agent-created resources, preserve reviewable evidence, and keep temporary QA artifacts out of commits.

Letting an Independent Agent Decide What to Verify

Implementers know what they intended to change and risk designing self-checks around their own implicit assumptions. For critical modifications, the author recommends delegating verification to a clean review agent that does not inherit the implementation conversation. Provide it solely with the original request, the actual diff, and the verification Skill, allowing it to independently determine the blast radius without relying on the implementer's screenshots, QA notes, or green-light assertions.

A concrete example from the original guide illustrates this: the requirement called for configuring the end-to-end test base URL via E2E_BASE_URL. The implementer tested against the default port 8080 and reported all tests passing. However, in e2e/global-setup.ts, seed image URLs remained hardcoded to http://localhost:8080/test-media/....

The independent reviewer launched the application on port 8081, immediately exposing 3 failed image requests directed at the old port and 3 fallback cards rendered on screen. A superficial check asserting only "cards exist" would still pass, as fallback cards are still cards. The fix required generating image URLs dynamically from the configured base URL or using appropriate relative paths.

This example demonstrates how an independent verification scope uncovers defects masked by default assumptions; it does not guarantee a second agent will catch every flaw.

Reusable Verification Skills: Two Equivalent browser-review Templates

The following two standardized workflow files are functionally equivalent, provided below as a translated English version of the guide's Chinese template and this guide's original English version. Teams do not need to use both; simply select one matching your workflow.

To use either version, create .agents/skills/browser-review/SKILL.md in the project root and save the full content (preserving the opening YAML frontmatter).

Translated Counterpart (English Translation of This Guide's Chinese Source)

The following block is the direct English translation of the custom Chinese skill defined in this guide. It serves as an equivalent counterpart to the original English reference below:

---
name: browser-review
description: Verify the runtime behavior of this task's user-visible changes in the real app and deliver reviewable evidence.
---

Perform browser verification for this task's user-visible changes:
1. First read the original requirements and actual git diff (including uncommitted edits and newly added files from this task), recording pre-existing work from git status --short. List affected routes, relevant loading/empty/error/selected/disabled states, adjacent paths, actions, expected outcomes, and verification methods. Run existing relevant tests and necessary build or type checks first, recording results.
2. Start the application using the project's dev script, README, or existing Docker configuration; wait for service readiness with a timeout before opening the browser. Retain logs and report if startup fails. Track all processes, containers, browsers, and temporary files you spawn. Do not disable authentication, edit .env, fake services, or add temporary routes bypassing real flows to simplify checks.
3. Complete affected user flows and inspect the final outcome. For saving, record initial values, modify them to distinct target values, and verify current values after reload. For resets, modify conditions first, then verify both inputs and result lists restore. Do not treat mere page loading or a click without an error as success.
4. Read DOM or accessibility snapshots for functional states; check relevant desktop and narrow viewports with screenshots for layout issues. When APIs are involved, inspect relevant responses and new errors; do not expand scope to pre-existing warnings or unrelated network traffic.
5. Upon finding defects, fix the minimal root cause, rerun relevant tests, and repeat the affected flow. Mark environmental hurdles such as missing documented credentials or unreachable preview servers as blocked. If an executed flow fails due to a real dependency API failure, preserve the failure and its cause. List unperformed checks as untested, explaining why.
6. Regardless of pass or fail, clean up only processes, containers, browsers, and temporary materials you spawned or created, preserving evidence needed for hand-off. Check git status --short again to safeguard pre-existing modifications and untracked files. Do not run git add, commit, push, or global cleanup commands.
7. Report routes, actions, expected and actual outcomes, supporting evidence, relevant test results, fixes, blocked/untested items, and cleanup status. Never mark checks lacking evidence as passed.

Original English Version (Standard Reference)

The block below is this guide's custom English Skill, preserved verbatim from the Chinese article:

---
name: browser-review
description: Verify this task's user-visible changes in the running app and report reviewable evidence.
---

Perform browser verification for this task's user-visible changes:
1. Read the original request and actual git diff, including this task's uncommitted changes and new files. Record pre-existing work with git status --short. Plan affected routes, relevant loading/empty/error/selected/disabled states, nearby flows, actions, expected outcomes and how to check them. Run existing relevant tests and any necessary build or type checks first, and record results.
2. Start the real app using its dev script, README or existing Docker configuration. Wait for readiness with a time limit; retain logs and report startup failure. Track only the processes, containers, browsers and temporary files you create. Do not disable authentication, edit .env, invent a fake service or add a bypass route to make verification easier.
3. Exercise each affected user flow and check the final result. Before saving, record initial values and set distinct target values; reload and read current values to check persistence. Before resetting, change the state, then check both inputs and results. Page loading or a click without an error is insufficient.
4. Use DOM/accessibility state for functional checks and relevant desktop/narrow viewport screenshots for layout. Inspect relevant responses and new errors when the flow uses an API. Do not expand scope to old warnings or unrelated requests.
5. Fix the smallest cause of a defect, rerun relevant tests and repeat the affected flow. Report missing documented credentials or an unreachable preview server as blocked. If an executed product flow fails because a real dependency API fails, keep the failure and its cause. List checks not performed and why.
6. Clean up your own processes, containers, browsers and temporary files on success or failure; preserve evidence needed for hand-off. Check git status --short again and preserve pre-existing edits and untracked files. Do not run git add, commit, push or global cleanup commands.
7. Report routes, actions, expected versus actual outcomes, supporting evidence, test results, fixes, blocked/untested items and cleanup. Do not claim a pass without evidence.

Connecting the Agent to a Browser: Playwright MCP or CLI + Skill

The workflow above requires a tangible, operational browser channel. Playwright automates browser control, element location, and state inspection, while MCP (Model Context Protocol) enables agents to invoke external tools. Playwright MCP exposes these browser operations directly to agents as structured tools rather than serving as a standard browser extension.

Microsoft's repository also outlines a Playwright CLI + Skill path: controlling browsers via command-line tools guided by on-demand documentation, which minimizes context window consumption during coding tasks. MCP, by contrast, provides persistent tool interfaces and stateful browser sessions. Either approach still relies on the verification rules outlined above.

The original guide ran on Node.js 20+; the official Playwright MCP repository specifies a minimum requirement of Node.js 18+. If installing katalepsis later in the workflow, Node.js 22.12+ is required.

Register the server according to your client. For Claude Code:

claude mcp add playwright npx @playwright/mcp@latest

For Codex:

codex mcp add playwright npx "@playwright/mcp@latest"

Reload the client connection and verify the browser tools appear. Then have the agent navigate to the TodoMVC Demo, add a "Browser QA" todo item, and confirm it renders in the list. This sanity check verifies channel connectivity; it does not replace testing your application.

How Does an Agent Locate Target Controls?

Playwright MCP can generate an accessibility snapshot: structuring the page into accessible roles, names, and states—such as button "Save" or textbox "Email"—along with element reference IDs (ref). The agent locates controls based on this live snapshot without guessing CSS selectors. When the page updates, the snapshot must be refreshed; agents cannot blindly click reference IDs copied from earlier logs or tutorials.

Semantic locators help center verification on user-observable behavior. Each test must also isolate its own data and state, avoiding reliance on leftover artifacts from prior checks. However, a button present in an accessibility snapshot does not prove it is unobstructed; verifying layout still requires inspecting screenshots.

Optional Tooling: Organizing Checks and Evidence with katalepsis

The verification workflow functions fully without katalepsis. The author initially used the Skill to guide direct browser interactions, later developing the MIT-licensed open-source CLI katalepsis to generate clean PR review artifacts. browser-review is the procedural Skill; katalepsis is the CLI tool for executing checks and compiling reports.

It writes output to artifacts/katalepsis/<run-id>/: an HTML report for browser review, a Markdown summary suitable for PR descriptions, and a manifest.json recording structured results alongside SHA-256 hashes of all evidence files. Screenshots, video recordings, and Playwright traces are collected according to execution paths, configurations, and test outcomes; users cannot assume every check automatically yields a complete evidence package.

Hashes can check file consistency; they do not guarantee files are tamperproof or prove that the correct test target was chosen. Traces record step-by-step actions, live DOM snapshots, and network exchanges to reconstruct failures, as detailed in the Playwright Trace Viewer documentation.

Two Execution Paths: run for Existing Tests, verify for Natural-Language Checks

verify assertions cover a narrower scope than standard Playwright test suites, lacking direct built-in assertions for input field values, persistent storage checks, and multi-viewport layout comparisons shown in earlier examples; specific boundaries are explained below.

Installation and Initial Setup

katalepsis is not yet published to npm. Its installer clones and builds from source, requiring Node.js 22.12+, Git, and pnpm 10 or corepack. The official installation command is shown below (this guide does not install or run it locally):

curl -fsSL https://www.katalepsis.glamboyosa.xyz/install.sh | sh

The installer places katalepsis and katalepsis-mcp into ~/.local/bin, which must be present in your terminal's PATH.

When existing Playwright tests are available, merge the following fields into the use block of your existing playwright.config.ts, retaining other settings:

use: {
  baseURL: process.env.KATALEPSIS_BASE_URL ?? "http://localhost:3000",
  screenshot: "only-on-failure",
  video: "retain-on-failure",
  trace: "retain-on-failure",
},

The project requires @playwright/test and active test suites. After launching the application, run the command from the project root, replacing the placeholder URL with your live local or preview address:

katalepsis run --cwd . --url http://127.0.0.1:3000

If the browser fails to launch due to missing binaries, append --install-browsers to the command to have the tool install the browser version matching the project's Playwright dependency. verify downloads its dedicated Chromium build upon its first run.

Verify how the base URL is actually consumed. The tool warns if the configuration omits KATALEPSIS_BASE_URL, but the presence of the environment variable does not guarantee individual tests actually utilize it. Hardcoded URLs such as page.goto("http://localhost:8080/login") will continue hitting the wrong server regardless. The author encountered this exact issue across two separate projects.

When no tests exist, create a checks.json file in the project root. The sample check below tests an empty form submission; /signup, button descriptions, and pass conditions should be customized to your application:

{
  "checks": [
    {
      "id": "empty-signup-validation",
      "name": "Empty signup form validation",
      "startPath": "/signup",
      "intent": "Submit the registration form without filling in any fields.",
      "passCondition": "Validation error messages for required fields are visible, and registration success confirmation is not displayed."
    }
  ]
}

Specifying both what should appear and what must not appear clarifies acceptance criteria. During report review, humans must still verify whether the model's assertions actually address that target.

Understand the data boundaries before running: verify transmits page DOM structures and text to the model provider, while screenshots are retained locally for report evidence rather than sent to the model. Text reports sanitize URL query parameters, but screenshots, video recordings, and traces are not automatically redacted. Always use test data appropriate for external transmission and sharing; on a real working branch, the author switched to synthetic seed data and followed his own permission rules before invoking model checks.

Taking OpenRouter as an example, configure credentials, select the provider, and query available model IDs:

katalepsis auth set openrouter
katalepsis config provider openrouter
katalepsis models openrouter
katalepsis config model 'REPLACE_WITH_MODEL_ID_FROM_ABOVE'
katalepsis verify --cwd . --url http://127.0.0.1:3000 --checks checks.json

As with run, start the application and substitute the real target URL. auth set stores keys in the system keychain, or they can be configured via supported environment variables; never pass credentials through command-line flags or MCP tool arguments.

Enabling Coding Agents to Run Checks and Ingest Reports

Running katalepsis as an MCP server allows coding agents to execute checks and read back reports autonomously after editing code. For Codex, replace the placeholder after --root with the absolute path of your project:

codex mcp add katalepsis -- katalepsis-mcp --root /path/to/your-app

Reload the client connection and confirm three tools appear: verification_run_tests (run existing tests), verification_check_browser (natural-language checks), and verification_get_report (retrieve reports). Project filesystem boundaries are governed by --root, while model authentication is configured via the preceding auth commands.

Browser checks typically run longer than standard tool calls. In Codex, append tool_timeout_sec = 600 inside the existing [mcp_servers.katalepsis] table in ~/.codex/config.toml; do not set this globally or overwrite existing launch arguments in that table. For other clients, consult the official MCP configuration docs.

Distinguishing the Four Report Verdicts

• passed: Meets all pass criteria for that execution path. If an existing Playwright test passes only after retries, it is recorded as passed while preserving a flaky flag and logging every attempt.
• failed: The check encountered an actual failure. If a real third-party API outage causes a flow to fail, it is still recorded as failed; this documents a runtime failure in this environment without automatically proving the root cause lies in the latest code edits.
• blocked: Unable to reach a valid verdict. Examples include network errors or 502/503/504 responses during preview health checks, failure to launch the browser, insufficient evidence submitted by the model, tool errors, or exhaustion of step, token, or time budgets.
• not tested: The check was not run, such as when filtered out or skipped. The tool lists these under notTested with an explanation and excludes them from passed counts.

When a preview environment is unreachable, the tool blocks affected checks prior to browser execution, preventing infrastructure downtime from generating false functional bug reports. Failures discovered during an executed flow must be judged against empirical evidence.

The Model Claims "Passed"—How Does the Engine Validate Evidence?

In verify, the model can call navigate, snapshot, click, fill, press, assert, and finish. Tool assertions are executed in a real browser, capturing screenshots, results, and assigned identifiers like a1. The model concludes by invoking finish with its verdict and referencing its assertion IDs.

The model's subjective text statement cannot self-certify. The programmatic arbiter function decideOutcome audits claims against execution logs:

1. References must map to genuinely executed assertions. Concluding without an explicit outcome, omitting references, or citing unrecognized assertion IDs results in a blocked verdict.
2. When claiming a pass, all referenced assertions must have passed, and at least one must be a visibility assertion (visible); confirming only a URL change or an element being hidden is insufficient.
3. If an unreferenced assertion fails after the cited assertions, the run cannot ignore the newer conflicting evidence to claim a pass.
4. Claiming a failure similarly requires citing assertions that genuinely failed; an unbacked claim of failure is likewise blocked.

Claims failing these evidential constraints are classified as blocked: no-evidence. This mechanism constrains the evidence submitted by the model; it does not automatically audit whether all business requirements are met.

Pre-existing Page Elements Cannot Independently Prove Action Success

The author encountered an instance where a model used "the email input label is visible" to claim form submission succeeded. Yet the label existed prior to the click; even though the assertion evaluated to true, it proved nothing about the outcome of the submission.

To prevent this, each check initializes separate browser contexts: the model operates on one page, while another loads the identical URL without performing any interactions, serving as an unmanipulated baseline control. When the model opens new URLs via navigate, corresponding baseline replicas are created as well.

When evaluating visibility assertions, the tool checks whether the identical element is also visible in these unmanipulated replicas, recording trueAtStart. For standard interaction checks, at least one passing visibility assertion must have been absent in the baseline replicas; if all cited visible elements existed from the start, the pass claim is rejected.

Checks inspecting initial page conditions can set "observeOnly": true on the check object in checks.json, bypassing the requirement for new visual changes while still requiring genuinely executed and passing visibility assertions. "Initial state" does not mean inspecting only above-the-fold content.

The baseline control issues an additional network request to the URL; browser context isolation does not equate to backend data isolation. Single-use activation links or endpoints that mutate server state upon access may yield divergent behavior on a second load, and a failed baseline request can cause the check to block.

An Assertion Can Be True Without Answering the Requirement

Suppose the specification requires "displaying an error message after submitting an empty form." Rather than asserting on the error message, the model clicks a standard link to "About Us" and asserts that the new heading is visible. This heading is genuinely new content that satisfies both visibility and baseline novelty gates, yet it has zero bearing on form validation.

For this reason, human reviewers must evaluate acceptance criteria, executed actions, and cited assertions in tandem. Verified browser assertions filter out unsubstantiated claims, but they cannot automatically guarantee relevance to requirements.

Currently, verify supports only three model assertion primitives: visible, hidden, and URL contains. The earlier profile example—inspecting live input values, toggle states, post-reload persistence, console logs, network payloads, and multi-viewport layouts—must be fulfilled by standard browser tools or Playwright test suites. This reflects the scope of katalepsis's current adapter layer rather than any inherent limitation of Playwright itself.

While evidential arbitration rules are deterministic, the actions chosen by the model vary across runs and model versions. Ephemeral toast notifications may dismiss before an assertion executes, requiring reviewers to examine screenshots, action logs, and traces when failures occur to distinguish application defects from uncaptured timing states. The project's repository documentation notes that external independent installation verification and cross-platform testing remain works in progress; the author's trial results should not be taken as universal reliability guarantees.

Verifying the Target URL and Confirming Preview Commit Provenance

Code may be modified while the preview deployment remains stale. A passing test on an outdated preview proves nothing about current changes. katalepsis inspects preview commit provenance in sequence, comparing it against the local checkout HEAD:

1. The x-commit-sha HTTP response header;
2. A <meta name="commit-sha"> tag within the initial 512KB of the HTML <head>;
3. A --preview-commit <sha> flag supplied by the caller.

The third option should only receive a known, actually deployed SHA, never an unverified pushed SHA or raw GITHUB_SHA. Results fall into three statuses:

• confirmed: Matches the checkout HEAD, allowing valid short SHA prefixes of at least 7 characters. This still does not prove uncommitted working tree modifications made it into the preview.
• unknown: Commit version cannot be determined; logged as unknown, and checks proceed.
• mismatched: Explicit mismatch. Checks still run, but runs that would have passed are converted to blocked; previously encountered failures are preserved.

Serializing Actions and Enforcing Check Resource Budgets

During testing, the author observed concurrent fill actions competing for focus, erroneously typing a name into an email field. The tool therefore queues browser actions serially, executing each action only after the preceding one completes; this does not mean network requests initiated by the page are serialized.

Each model check operates in an isolated browser context with tracing enabled, capped by default at 15 steps, 200,000 tokens, and 120 seconds; hitting these budget limits marks the run blocked with an explanation. Top-level navigation is restricted to the preview origin and origins explicitly permitted via --allow-origin (matching protocol, host, and port). External subresources can still load, so this does not constitute a full network sandbox.

Deliver Verifiable Runtime Outcomes Before Declaring Done

This verification methodology targets user-observable runtime behaviors: forms, filters, modals, navigation, responsive layouts, and post-reload state persistence. Fixing documentation typos or adjusting backend logic without UI implications does not warrant launching a browser; select the appropriate verification method for the change at hand.

Existing automated tests, browser verification, code review, and human inspection of critical changes each serve distinct roles. Returning to the initial settings page: an agent's final hand-off should clearly state what values it changed, how it saved and reloaded, what runtime state it observed, what issues it encountered, and where reviewable evidence is stored. This kind of hand-off can reduce the back-and-forth of humans taking screenshots, relaying errors, and prompting agents to retry.