CLI Agent Evals

Unified live eval toolkit for coding-agent CLIs.

cli-evals runs the same scenario suite across Claude Code, OpenAI Codex, opencode, and GitHub Copilot by driving real interactive CLIs through @agent-ix/agent-pty.

Setup

If this plugin is uninitialized or a command fails, follow the plugin setup guide for its required CLIs, configuration, and local diagnosis.

Community help

If the setup checks leave a reproducible Agent IX cli-agent-evals bug that blocks progress, join the Agent IX Discord. Community help is a last resort for Agent IX product bugs, not a help desk for local credentials, machine setup, third party tools, or unrelated projects. See setup.md for what to include.

Install

pnpm add -D @agent-ix/cli-agent-evals

Or install globally:

npm install -g @agent-ix/cli-agent-evals

Prerequisites for live runs:

• tmux >= 3.0
• claude, codex, opencode, or Copilot/gh on PATH
• authenticated local agent sessions for the agents you run

Check local readiness:

cli-evals doctor

Quickstart

Create cli-agent-evals.config.mjs:

import { defineSuite } from "@agent-ix/cli-agent-evals";

export default defineSuite({
  name: "my-cli",
  rootDir: import.meta.dirname,
  scenarios: [
    {
      id: "EV-001",
      canary: true,
      prompt:
        "Use my CLI to complete the happy path, then print the success sentinel.",
    },
  ],
});

Run canaries:

cli-evals run --suite ./cli-agent-evals.config.mjs --canary --agent claude --model sonnet

Run one scenario in release-evidence mode, retaining its workdir and captured transcript:

cli-evals run --suite ./cli-agent-evals.config.mjs --filter EV-001 --agent codex --model gpt-5 --keep

Print a previous report:

cli-evals rebuild --report evals/reports/latest.json

Plugin Install

Claude Code:

claude plugin marketplace add agent-ix/agent-plugins
claude plugin install cli-agent-evals@agent-ix

Codex:

codex plugin marketplace add agent-ix/agent-plugins
codex plugin add cli-agent-evals@agent-ix

opencode:

gh skill install agent-ix/cli-agent-evals --all --scope user --agent opencode

GitHub Copilot:

gh skill install agent-ix/cli-agent-evals --all --scope user --agent github-copilot

Installed skills:

• cli-evals: run, inspect, and debug suites.
• cli-evals-author: draft scenarios, fixtures, assertions, and classifiers.

Docs

• Usage
• Suite authoring
• Agent drivers
• Reports
• Plugin install