How to Systematically Evaluate Agent Skills: A Complete Guide from Trigger Cases to Automated Scoring

When developing Agent Skills, the most common frustration is relying on gut feel to judge changes: you swap in a new prompt version, and certain cases appear smoother, but it is hard to verify whether things genuinely improved, and even harder to spot regressions—scenarios that previously triggered stop responding, crucial execution steps get skipped, or temporary files are left behind in the workspace. The way out of this guesswork is establishing a comparable evaluation mechanism (Evals), using repeatable, comparable execution records to judge performance shifts and catch regressions promptly.

A Skill is essentially an organized collection of prompts and instructions. Evaluating a Skill is akin to lightweight end-to-end integration testing for agents: input target Prompts → fully capture the Agent's execution process (Traces and generated artifacts) → run targeted rule checks → output scores that can be compared across versions.

Putting evaluations into practice breaks down into six steps: define success criteria, write an evaluable Skill, run manual trials to expose hidden assumptions, build a 10–20 case mini-eval suite with positive and negative samples, use JSONL and deterministic graders to inspect behavior and artifacts, and use read-only Codex with JSON Schema for rubric-based grading. Building on this foundation, advanced monitoring can then be introduced as the setup matures.

Step 1: Define Success Criteria

The root reason many Skills are hard to evaluate is that "successful execution" was never clearly defined at the start. With only vague expectations, final acceptance often comes down to arbitrary manual spot-checking. The first step is to define success criteria clearly, breaking them down into concrete dimensions focused on must-pass requirements.

In practice, before writing or refactoring, compile a concise checklist that clearly distinguishes four goal dimensions:

• Outcome goals: Was the task ultimately completed? Does the built application run properly?
• Process goals: Did the Agent actually invoke the intended Skill? Did it execute the specified tools and commands in the expected order?
• Style goals: Do the generated file structure, component patterns, and styling conventions match established technical specifications?
• Efficiency goals: Were there wasteful retries during execution (such as repeatedly attempting the same failed command, or consuming an unusually large number of Tokens)?

The standard for passing is whether all hard requirements are met. For example, in a scaffolding scenario, "was npm install executed" and "was package.json generated" are hard process and outcome checks that must pass strictly; meanwhile, "is the directory structure clean" and "does styling follow constraints" are checked item by item as style criteria.

After completing this step, you will have a structured acceptance checklist covering all four dimensions. It provides a clear target for writing subsequent test assertions, as the core purpose of evals is to expose concrete regression signals early rather than produce an ambiguous, blended pass rate.

Step 2: Write an Evaluable Skill

For evaluations to work, the Skill itself must have clear, verifiable behavioral boundaries. The second step is writing a standardized Skill skeleton that contains explicit acceptance criteria.

In the Codex ecosystem, a Skill is essentially a self-contained directory with a SKILL.md file. The top of the file must include YAML Front Matter consisting of name and description, followed by Markdown instructions that guide Agent behavior, along with optional supporting scripts or resources.

Here, name and description are the most critical trigger signals for the entire Skill. Codex relies on these two metadata fields to determine whether the current task should invoke the Skill and decide when to inject the full contents of SKILL.md into the context window. If the name and description are overly broad or semantically overloaded, the Skill will fail to activate reliably.

If starting from scratch, you can invoke the official scaffolding skill directly in your terminal:

$skill-creator

The scaffolding tool interactively guides you to clarify the Skill's responsibilities, trigger conditions, and whether it should be instruction-only or script-backed. The default recommendation is to favor an instruction-only design first.

Practical Example: A Standardized Demo Scaffolding Skill

Take a minimal practical Skill named setup-demo-app as an example. Its responsibility is to reliably initialize a lightweight React demo project under consistent standards: use Vite's React + TypeScript template, configure Tailwind CSS via the official Vite plugin, follow a lean directory layout, and provide a directly verifiable Definition of Done.

Depending on the desired scope, you can save the content below to one of two paths:

• Repo-scoped: .codex/skills/setup-demo-app/SKILL.md
• User-scoped: ~/.codex/skills/setup-demo-app/SKILL.md

---
name: setup-demo-app
description: Scaffold a Vite + React + Tailwind demo app with a small, consistent project structure.
---

## When to use this

Use when you need a fresh demo app for quick UI experiments or reproductions.

## What to build

Create a Vite React TypeScript app and configure Tailwind. Keep it minimal.

Project structure after setup:

- src/
  - main.tsx (entry)
  - App.tsx (root UI)
  - components/
    - Header.tsx
    - Card.tsx
  - index.css (Tailwind import)
- index.html
- package.json

Style requirements:

- TypeScript components
- Functional components only
- Tailwind classes for styling (no CSS modules)
- No extra UI libraries

## Steps

1. Scaffold with Vite using the React TS template:
   npm create vite@latest demo-app -- --template react-ts

2. Install dependencies:
   cd demo-app
   npm install

3. Install and configure Tailwind using the Vite plugin.
   - npm install tailwindcss @tailwindcss/vite
   - Add the tailwind plugin to vite.config.ts
   - In src/index.css, replace contents with:
     @import "tailwindcss";

4. Implement the minimal UI:
   - Header: app title and short subtitle
   - Card: reusable card container
   - App: render Header + 2 Cards with placeholder text

## Definition of done

- npm run dev starts successfully
- package.json exists
- src/components/Header.tsx and src/components/Card.tsx exist

Through explicit step breakdowns and a Definition of done, this step yields a well-structured SKILL.md file. Judging whether it passes comes down to whether output boundaries and completion conditions are clear and verifiable; if rules are vague or shifting, subsequent automated evaluations will lack a baseline for judgment.

Step 3: Run Manual Trials and Expose Hidden Assumptions

After writing SKILL.md, avoid rushing into writing extensive scripts. This step is about running manual trials in a real or temporary sandbox environment to observe where the Agent diverges from expectations, thereby surfacing and resolving unstated assumptions in the Skill.

You can start in the interactive interface by selecting the skill via the /skills slash command, or by referencing it explicitly in your prompt with $setup-demo-app. The priority in this phase is systematically investigating three common types of hidden assumptions:

• Triggering assumptions: General requests that should trigger (e.g., "help me spin up a quick React demo") fail to match; or adjacent scenarios that should not trigger (e.g., asking within an existing project to "help me configure Tailwind for an existing app") get accidentally activated.
• Environment assumptions: The Skill assumes it is running in a completely clean, empty directory, or assumes the operating system definitely has npm installed and prioritizes it over other package managers.
• Execution assumptions: The Agent mistakenly assumes dependencies are already installed globally and skips npm install, or rushes into configuring Tailwind before the Vite project has finished initializing.

When you are ready to formalize these findings, switch from the interactive interface to the headless CLI tool codex exec. This command streams execution progress to standard error (stderr) while directing only final output to standard output (stdout), making it ideal for script orchestration and CI integration.

Because scaffolding involves writing and creating disk files, you must explicitly supply the --full-auto permission flag during execution:

codex exec --full-auto \
  'Use the $setup-demo-app skill to create the project in this directory.'

When invoking automated runs, adhere to the principle of least privilege: grant only the minimum scope of permissions required to complete the task, and verify that no unexpected privilege escalation occurs. The goal of this step is not error-free first runs, but documenting where behavior diverged from expectations. Once the investigation is complete, you will have a set of boundary issues and repeatable invocation commands that form the core source for the next step's eval suite.

Step 4: Build a 10–20 Case Mini-Eval Suite with Positive and Negative Samples

Evaluating an Agent does not require stacking up hundreds or thousands of benchmark cases from day one. For an individual Skill, maintaining a mini-eval suite of 10 to 20 representative cases is sufficient to catch valuable regression signals early.

This step turns the boundary issues surfaced earlier into formal test cases. You can create a compact CSV file (e.g., evals/setup-demo-app.prompts.csv) to track trigger expectations and concrete Prompts in tabular format, appending new rows as fresh issues emerge:

id,should_trigger,prompt
test-01,true,"Create a demo app named `devday-demo` using the $setup-demo-app skill"
test-02,true,"Set up a minimal React demo app with Tailwind for quick UI experiments"
test-03,true,"Create a small demo app to showcase the Responses API"
test-04,false,"Add Tailwind styling to my existing React app"

These four cases represent four indispensable testing intents:

• Explicit invocation (e.g., test-01): The prompt explicitly includes $setup-demo-app. It verifies that the underlying invocation pipeline works when the user names the skill directly, ensuring later configuration changes do not break basic activation.
• Implicit invocation (e.g., test-02): Omits the Skill name entirely, stating only the standard use case targeted by the skill. It verifies whether name and description in the Front Matter are clear enough for the model to choose the skill autonomously in everyday conversations.
• Contextual invocation (e.g., test-03): Introduces a specific business scenario (such as integrating the Responses API), adding mild real-world noise. It tests whether the model can still accurately extract scaffolding needs and maintain consistent code structure in the presence of extraneous details.
• Negative control (e.g., test-04): Defends against false positives. The user asks to "add Tailwind styling to an existing project"—semantically close to the Skill description, but it must not trigger new project scaffolding. Lacking negative controls frequently causes Agents to overwrite existing projects when users simply intend to modify existing code.

By pairing positive implicit invocations with negative controls, the passing standard is straightforward: positive scenarios activate reliably with complete steps, and negative scenarios firmly avoid false triggers. Completing this step leaves you with a structured evaluation dataset file (evals/setup-demo-app.prompts.csv).

The planner generator below turns this step into a ready-to-use template: enter your Skill name, target task, required command, and required file, and it generates four categories of test Prompts, a deterministic checklist, and suggested file paths.

Step 5: Inspect Behavior and Artifacts with JSONL and Deterministic Graders

To verify conclusively that an Agent truly executed underlying steps rather than merely happening to produce seemingly usable code, you cannot rely on subjective guesswork. This step uses deterministic graders to inspect command traces and disk artifacts directly.

The key to achieving this is enabling the --json flag when running codex exec, which converts standard output (stdout) into a line-delimited JSONL stream of structured events. Every underlying Shell command invocation is clearly recorded via item.started and item.completed events, with item.type set to command_execution.

Automated Execution and Inspection Script

Leveraging this mechanism, we can write a lightweight Node.js script, evals/run-setup-demo-app-evals.mjs. Running this script sequentially executes the Agent, captures the trace, parses JSONL events, and asserts directly on command records and disk file existence:

// evals/run-setup-demo-app-evals.mjs
import { spawnSync } from "node:child_process";
import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs";
import path from "node:path";

function runCodex(prompt, outJsonlPath) {
  const res = spawnSync(
    "codex",
    [
      "exec",
      "--json", // Key flag: output a structured JSONL event stream
      "--full-auto", // Grant the required filesystem modification permissions
      prompt,
    ],
    { encoding: "utf8" }
  );

  mkdirSync(path.dirname(outJsonlPath), { recursive: true });

  // In --json mode, stdout contains standard JSONL-formatted content
  writeFileSync(outJsonlPath, res.stdout, "utf8");

  return { exitCode: res.status ?? 1, stderr: res.stderr };
}

function parseJsonl(jsonlText) {
  return jsonlText
    .split("\n")
    .filter(Boolean)
    .map((line) => JSON.parse(line));
}

// Deterministic rule 1: Did the Agent actually run a command containing npm install?
function checkRanNpmInstall(events) {
  return events.some(
    (e) =>
      (e.type === "item.started" || e.type === "item.completed") &&
      e.item?.type === "command_execution" &&
      typeof e.item?.command === "string" &&
      e.item.command.includes("npm install")
  );
}

// Deterministic rule 2: Was package.json actually created on disk?
function checkPackageJsonExists(projectDir) {
  return existsSync(path.join(projectDir, "package.json"));
}

// Single-case evaluation flow
const projectDir = process.cwd();
const tracePath = path.join(projectDir, "evals", "artifacts", "test-01.jsonl");

const prompt =
  "Create a demo app named demo-app using the $setup-demo-app skill";

// Step 1: Invoke the CLI and persist the trace to disk
runCodex(prompt, tracePath);

// Step 2: Read the trace events and run assertions
const events = parseJsonl(readFileSync(tracePath, "utf8"));

console.log({
  ranNpmInstall: checkRanNpmInstall(events),
  hasPackageJson: checkPackageJsonExists(path.join(projectDir, "demo-app")),
});

The pass criteria for these deterministic checks are straightforward: assertions like ranNpmInstall and hasPackageJson in the script output must all return true. After running, you receive a full trace artifact saved at evals/artifacts/test-01.jsonl along with unambiguous boolean check results. If a check fails, developers do not need to guess; they can open the .jsonl trace log directly and inspect recorded command execution events in chronological order.

Step 6: Use Read-Only Codex and JSON Schema for Rubric-Based Grading

Deterministic code assertions effectively confirm whether fundamental actions occurred, but struggle to evaluate whether generated output conforms to code style conventions. For example: did the code use functional components exclusively? Did Tailwind strictly use utility classes rather than mixing in CSS Modules? Is the component directory structure properly organized?

This step introduces a second, rubric-based model review pass using predefined criteria after deterministic checks. Specifically, use the --output-schema flag of codex exec to constrain the model to structured review data conforming to a specified JSON Schema, conducting a read-only review without modifying code.

Define the Evaluation Schema

First, create an evaluation rubric contract file at evals/style-rubric.schema.json. In this Schema, the top level includes overall pass status (overall_pass) and an overall numeric score (score), while individual review dimensions consist of item-by-item pass decisions (pass) and evaluation notes (notes):

{
  "type": "object",
  "properties": {
    "overall_pass": { "type": "boolean" },
    "score": { "type": "integer", "minimum": 0, "maximum": 100 },
    "checks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": { "type": "string" },
          "pass": { "type": "boolean" },
          "notes": { "type": "string" }
        },
        "required": ["id", "pass", "notes"],
        "additionalProperties": false
      }
    }
  },
  "required": ["overall_pass", "score", "checks"],
  "additionalProperties": false
}

Execute Read-Only Code Review

Once scaffolding completes, invoke codex exec targeting that directory once more. This time, omit the --full-auto permission flag, supplying only the review prompt and Schema path, and save the scoring JSON into the artifacts directory:

codex exec \
  "Evaluate the demo-app repository against these requirements:
   - Vite + React + TypeScript project exists
   - Tailwind is configured via @tailwindcss/vite and CSS imports tailwindcss
   - src/components contains Header.tsx and Card.tsx
   - Components are functional and styled with Tailwind utility classes (no CSS modules)
   Return a rubric result as JSON with check ids: vite, tailwind, structure, style." \
  --output-schema ./evals/style-rubric.schema.json \
  -o ./evals/artifacts/test-01.style.json

Review results are written into structured JSON: overall_pass indicates overall pass status, score supports cross-version comparisons, and each check item provides pass and notes. If teams establish their own score thresholds, corresponding assertions can be added to the eval script; the original example does not prescribe a universal threshold. This step outputs an evaluation report file at evals/artifacts/test-01.style.json.

Outputs constrained by --output-schema are no longer freeform natural language that is difficult to parse automatically; they are structured data that can be merged, compared, and tracked directly within CI or automated evaluation pipelines. When running in GitHub Actions, the official Codex Action natively supports passing this flag via codex-args.

Advanced Monitoring for Maturing Systems

Once the foundational two-layer "execution-assertion-scoring" loop runs reliably, the evaluation system can expand as skill complexity grows. Prioritize fast, decisive behavioral feedback first, gradually layering in heavier, more time-consuming verification methods as testing deepens.

The following six directions can progressively extend basic evals into more realistic engineering scenarios:

• Command thrashing and loop retry checks: Count the occurrence frequency of command_execution in JSONL logs. If an iteration causes a surge in identical command executions, the Agent is likely trapped in an error-retry loop.
• Token consumption monitoring: Extract usage.input_tokens and usage.output_tokens from turn.completed events. This directly flags prompt bloat or context redundancy, preventing single-run costs from spiraling out of control.
• Static compilation validation: Have the evaluation system execute npm run build after the Agent completes its run. This serves as a critical end-to-end hard gate, effectively catching subtle syntax errors, broken module imports, or toolchain version mismatches.
• Runtime smoke detection: Run npm run dev to start the server, then query the local port with curl or run a minimal Playwright script to verify that the UI mounts cleanly. Because this testing is time-consuming, enable it selectively for high-risk core cases.
• Repository cleanliness checks: Execute git status --porcelain to assert that no unexpected stray configuration files or temporary code remain in the workspace after execution, keeping the codebase clean.
• Sandbox and permission regression testing: Verify whether the Skill continues to operate smoothly in a basic restricted sandbox without elevated privileges, confirming that iterations introduce no unexpected privilege escalations or commands that forcibly require elevated permissions.

Conclusion

Moving from gut-feel tweaks to retaining traces, itemized checks, and continuous comparison grounds Agent Skill iterations on solid engineering fundamentals. This workflow replaces the subjective guess of "this version feels smoother" with structured, factual evidence:

Start by defining clear success criteria and authoring a standardized Skill; uncover boundary assumptions through manual trial runs and consolidate debugging discoveries into 10–20 representative eval cases; at the base layer, validate commands and file artifacts using JSONL event streams and deterministic graders, while using JSON Schema at the upper layer to standardize model-driven code reviews; and as the skill matures, selectively expand into compilation checks and runtime monitoring.

With a repeatable evaluation foundation in place, every prompt adjustment or logic tweak clearly reveals what improvements were made while immediately exposing unintended regression signals. By continually expanding test cases with real-world issues, a Skill can reliably maintain its capabilities across long-term iteration.