arthai-marketplace

/eval

Run a model/stack eval study on your own tasks — wizard by default, quick for a zero-decision smoke contrast, deterministic grading, and a report that leads with a plain verdict.

Synopsis

/eval                                  # interactive wizard (the default)
/eval quick                            # canned 3-task set, current stack vs claude-only, one confirm
/eval run --set <set> --arms <csv> [--reps N] [--ceiling USD] [--dry-run]
/eval status                           # cells done, spend vs ceiling, failures
/eval report                           # plain verdict first, tables underneath
/eval report --chain                   # full-chain studies: per-stage drill-down + judge lane
/eval run --from-issues owner/repo#12,#34 ...   # scaffold tasks from bug reports

When to use it

You mixed a cheaper model into your stack (via /arth-setup) and want to know whether it silently breaks things while appearing to succeed.
You want a repeatable, versioned comparison — same task set, different arms, across time (including before/after a toolkit release).
You want pass@3 evidence for “capable but inconsistent” vs “incapable” before changing your model policy.
Not for: day-to-day “did my change work?” checks — that’s /qa. Not for adding providers or keys — that’s /arth-setup (eval validates, it never installs).

Quickstart

/eval quick

What you’ll see: a preflight summary, one confirmation showing the matrix (3 tasks × 2 arms) with a cost estimate and a spend ceiling, then a background run. When it finishes, /eval report opens with a one-paragraph verdict like “claude passed 3 of 3 tasks at $0.41 per passing task — deepseek was cheaper but failed more.”

Examples

/eval                                   # wizard: pick set → arms → reps → confirm ceiling
/eval quick                             # zero decisions, one confirm
/eval run --set smoke --arms claude,deepseek --reps 3 --ceiling 10
/eval run --set smoke --arms claude --dry-run     # estimate only, no API calls
/eval status                            # progress of the background run
/eval report                            # verdict, leaderboard, variance strip, drill-down

Arguments & flags

Flag	Values	Default	What it does
`--set`	a set under `tests/eval/tasks/sets/`	—	which named, versioned task set to run
`--arms`	`claude,<provider>[:<model>]`	—	arms to compare; every provider arm must hold a fresh conformance pass
`--reps`	integer	1	repetitions per task (3 gives you pass@3)
`--ceiling`	USD	—	hard spend abort threshold; required for any paid matrix, always the last question
`--dry-run`	—	off	resolve the matrix and estimate cost with zero API calls and zero writes
`--resume`	run id	—	skip cells already completed in the journal
`--from-issues`	`owner/repo#N,#M`	—	scaffold task specs from issue title/body (never from fix diffs)
`--config-arm`	`bare-claude` / `toolkit` / `toolkit+memory` / `toolkit+jev` / `toolkit+memory+jev`	`toolkit`	for sets that declare `config_arms`: run the same tasks bare (same goal text, one invocation, no toolkit), with the toolkit chain, with the toolkit plus an isolated per-cell memory store, or with the decisions product enforced inside each cell (`+jev`; needs `TYPESAFE_API_KEY`)
`--permission-mode`	`acceptEdits` / `dontAsk` / `bypassPermissions`	`acceptEdits`	the Claude CLI permission mode inside each cell; chain studies whose stages run tests and git need `bypassPermissions` (the checkout is frozen and has no remote); recorded on the run
`compare --baseline <run> --candidate <run>`	two run ids, one change apart	—	paired pass and cost deltas; `toolkit → toolkit+jev` adds the Jev sentence
`regrade --run <run> [--cell <cell>]`	a finished run id	—	re-grade finished cells from their saved diffs under the current grading rule; appends a `regrade` record, never rewrites the original result (`scripts/eval-runner.sh regrade …`)
`report --chain`	—	off	chain-study report: per-task × per-arm × per-stage drill-down, reviewer-findings counts, and the judge lane

What it does

Preflight — validates everything, installs nothing on your stack. Eight checks in a fixed order: arm legality (fresh probe pass required — an unprobed or failed model is refused with the exact probe command), remote liveness, local readiness, proxy coverage, workspace capacity, grader readiness, telemetry honesty, and the spend gate last. Grader readiness prepares the set’s grading environment and measures the untouched base commit (see How grading works); a grader that cannot run is refused here, before any money is spent. The one exception to “installs nothing”: a missing local model can be pulled, but only after a size-stated confirmation.
Confirmation checkpoint. The resolved matrix, the cost estimate, and the spend ceiling are shown in one final prompt — nothing runs and nothing is spent until you confirm. The equivalent /eval run … one-liner is printed so the study is repeatable.
The run. One frozen checkout per cell (task × arm × rep) at a pinned commit with history truncated — the agent cannot look up the fix. Each cell is a non-interactive session, graded on exit by the task’s own deterministic checks (hidden tests are copied in only after the agent exits). Every cell is journaled to tests/eval/runs.jsonl; interrupting is safe, --resume skips completed cells. Costs are recorded at real descriptor pricing, and the run hard-aborts if spend crosses the ceiling.
Grading. Pass means the task’s acceptance checks go green and the change introduced no new failures compared with the untouched base commit — no AI judging anywhere in the pass rate. Cheat-pattern signals (weakened tests, swallowed exceptions, widened lint ignores) are recorded per cell for you to review, but never change the pass/fail outcome on their own.

How grading works

Real codebases are rarely spotless at the commit you pin: a linter already reports a few dozen findings, the type checker a few hundred, some tests need a database you don’t have locally. A rule of “everything must be green” would fail every arm for reasons that have nothing to do with the work. So /eval grades the difference.

A prepared grading environment. A set can declare grader_setup — the project’s own install commands (create a virtualenv and install its requirement files, npm ci, …). They run once against a pristine checkout of the pinned commit, into ~/.arthai/eval/envs/ — outside your project and outside every cell’s checkout — and are cached, kept read-only, and re-verified before each grade. If a setup command fails, preflight refuses the run and shows its output.
A baseline at the pinned commit. Preflight runs every task’s checks once at the untouched base and caches the result. You see it per task: green at base, or RED at base … ruff 36, mypy 605, pytest 156 (184 passing).
Differential gates. For ruff, mypy, pytest, tsc and jest the engine compares structured results, ignoring line numbers. A gate fails only when the change adds a new failure (a second copy of an existing one counts), or when a test that passed at base is now deleted, skipped or marked expected-to-fail. Pre-existing failures are not held against any arm; fixing them is reported but never required. Every failing gate names the new failures.
No silent outcomes. A check the engine cannot parse keeps the simple rule (exit 0) and is allowed only if it already passes at base — otherwise preflight refuses that task by name and tells you to narrow the command. Acceptance checks (a hidden test that is supposed to fail before the fix) are exempt by design.
Same tools for every arm. With agent_env: true (the default when a set declares grader_setup) the same prepared environment is on the agent’s PATH in every arm, so each arm can actually run the project’s tests while it works. The choice is recorded on the run, and compare refuses two runs that differ in it.

Regrading a finished run. Every cell’s final diff is saved, so results graded under an older rule can be re-graded without re-running (or re-paying for) the agents: scripts/eval-runner.sh regrade --run <run-id> rebuilds each cell’s tree at the run’s pinned commit, applies the saved diff strictly, and grades it with the current gates. The original result stays in the journal untouched; report, status and compare use the newest regrade and say so (regraded 2026-09-19: differential-v1 gates …).

Full-chain studies

Some task sets (for example a set built from a real project’s open issues) define each task as a complete workflow chain — /planning → /implement → /qa → /fix (only when /qa fails) → /pr → /review-pr — instead of a single prompt. For those sets:

Every stage is captured separately (cost, wall-clock, turns, exit), so the report shows not just which arm won a task but which workflow stage diverged (an arm that plans fine but thrashes in /fix is visible as exactly that).
The /review-pr stage’s findings are counted per cell (with severities) as a deterministic quality column — findings are enumerable artifacts, not AI opinions.
Draft PRs only, never merged. The chain’s /pr stage opens a draft PR labeled eval:<set>:<task>:<arm> against the project under test. The runner refuses task specs whose prompts contain a merge command, and any merge attempt detected at runtime is journaled as an error that invalidates the cell. Adopting any fix is your decision after the study — the comparison table doubles as your pick list.
config_arms lets one set answer “what is the toolkit worth?”: the bare-claude arm gets the same goal text as a single plain invocation (equal information, no toolkit), toolkit runs the chain, toolkit+memory adds an isolated per-run memory store, and toolkit+jev / toolkit+memory+jev run the same chain with the decisions product enforced inside each cell (its decision log is kept per cell and copied beside the cell’s diff).
Cells are blind. Each cell works in a frozen checkout with no remote. If a cell’s output shows it reading pull requests or issues, fetching from git, opening the study’s own journal or artifacts, or touching another cell’s workspace, the runner journals a peek attempt and invalidates that cell — the same posture as a merge attempt.

/eval report --chain renders the per-stage view. Sets that pin base_sha: TBD-at-launch are refused by preflight until the launch step fills in the commit — authoring a set and running it are deliberately separate moments.

The judge lane (optional, separate axis)

For chain studies you can add a calibrated qualitative comparison beside the deterministic verdict — it never changes pass/fail:

scripts/judge-lane.sh packets   --run <run-id>          # blind A/B diff packets
scripts/judge-lane.sh collect   --run <run-id> --verdicts my-verdicts.jsonl
scripts/judge-lane.sh calibrate --run <run-id> --judge-id <model>@<YYYY-MM-DD>

Packets are pairwise and blind: arm identity is stripped and A/B order is randomized by a recorded seed. You (the human) rate a calibration subset first; a judge model’s verdicts only unlock a report column when it agrees with your calibration verdicts at 0.8 or better, and a judge is never allowed to rate its own model family’s output. Below the bar, the column is refused outright — no “directional” scores. When calibrated, /eval report --chain shows the judge’s pairwise preference rates as a clearly separated section.

Jev as a rater (optional). scripts/eval-annotate.sh judge --run <run-id> lets the decisions product rate the same blind packets as one more rater. Its verdicts go through the same collect and calibrate steps and face the same 0.8 bar — and it is refused outright for runs whose config arm includes jev, so Jev never grades work Jev steered.

Testing with and without decisions

Run the same set twice, one arm at a time, then compare:

/eval run --set smoke --arms claude --reps 1 --ceiling 5 --config-arm toolkit
/eval run --set smoke --arms claude --reps 1 --ceiling 5 --config-arm toolkit+jev
/eval compare --baseline <first-run> --candidate <second-run>
/eval report --run <second-run>

compare prints the paired pass and cost deltas with one sentence about the decisions Jev made; report adds the Jev columns below. For your own repository, scaffold tasks from bug reports (/eval scaffold … --leak-check), declare config_arms in the set, and pass --permission-mode bypassPermissions for chain stages.

Jev columns and annotations

When a run uses a +jev config arm, /eval report adds one block after the leaderboard — Jev columns — with the decision layer’s own numbers: decisions made, what they cost, how often Jev landed in the uncertain band (escalation), p50 decision latency, and how often it agreed with the toolkit’s existing logic on the same decision. The verdict paragraph gains one sentence with those numbers. Pass/fail never reads any of it. /eval compare --baseline <run> --candidate <run> accepts one-change pairs only (for example toolkit → toolkit+jev) and prints the pass and cost deltas next to that Jev sentence.

Each graded +jev cell also stamps its verdict onto the decisions it made (a proxy label, kept separate from the labels the skills write themselves), so the decision log can be joined back to what actually happened. Proxy labels never count toward a gate’s calibration unless you ask for them explicitly.

Two more auxiliary views exist as separate files, never blended into the verdict: scripts/eval-annotate.sh markers ranks the recorded cheat markers for your audit, and failures classifies failed cells (provider failure, harness error, task defect, real failure). /eval report --annotations <file> prints them as their own section. Both are refused for any run whose config arm includes jev. /eval scaffold … --leak-check adds one advisory line saying whether a task goal names its own fix.

Output & artifacts

tests/eval/runs.jsonl — the append-only journal (system of record, machine-local, not committed)
/eval report — plain verdict first, then: leaderboard (pass rate, effective cost per passing task, latency with cloud and local kept separate), Jev columns when a +jev arm ran, a variance strip per task so unstable measurements can’t be over-read, and a per-cell drill-down (gates, cheat markers, resolved endpoint and model)
~/.arthai/eval/artifacts/<run-id>/ — each cell’s final diff (<cell>.diff, plus <cell>.untracked.diff for files it created but never staged) and, for +jev cells, its decision log (<cell>.jev.jsonl; a regrade adds <cell>.regraded.jev.jsonl)
~/.arthai/eval/envs/ and ~/.arthai/eval/baselines/ — the cached grading environments and per-task baselines at the pinned commit (safe to delete; the next preflight rebuilds them)
Scaffolded task specs under tests/eval/tasks/, hidden tests under tests/eval/fixtures/<task-id>/
Runs are tagged arth.experiment=eval:<set>:<task>:<arm>:<rep> so the Arth Intelligence experiments view groups them automatically when telemetry is on

Troubleshooting

Problem	Fix
`no conformance record — refused. Run: scripts/provider-registry.sh probe <name>`	The arm was never probed. Run the printed probe command (or re-run `/arth-setup`), then retry.
`conformance pass is stale … re-probe`	Probe verdicts expire after 14 days. Re-run the printed probe command.
`key <NAME> not present`	Add the key through `/arth-setup` — eval never collects keys itself.
`grader_setup[<id>] failed (exit N)`	The set’s own install command failed — the refusal includes its output (a missing interpreter such as `python3.11`, no network, a bad requirement). Fix that and re-run preflight; nothing was spent.
`… is broken at base — <tool> exited N with no parseable result`	The check cannot run in the grading environment (typically a missing dependency). Add the missing install step to the set’s `grader_setup`.
base is red for <task> and `<cmd>` is not differential-capable	That check already fails at the pinned commit and is not one of ruff, mypy, pytest, tsc or jest, so a fair verdict is impossible. Narrow the command to one of those tools, or fix it at the base.
A run graded before differential grading shows every cell as failed	Re-grade it from the saved diffs: `scripts/eval-runner.sh regrade --run <run-id>`, then `/eval report`.
`mixed grading rules across the pair` from `compare`	One run was regraded and the other was not. Regrade both.
First preflight of a set takes minutes	It is building the grading environment and baselines once; later preflights are cache hits. `--dry-run` never builds anything.
`estimate exceeds the ceiling`	Shrink the matrix (fewer arms/reps) or deliberately raise the ceiling at the confirm step.
Run interrupted mid-matrix	Nothing is lost — re-run with `--resume <run-id>` (shown by `/eval status`); completed cells are skipped.
`telemetry was dark for this run` in the report	The intelligence engine was unreachable during the run. Results are complete (the journal is the system of record); only dashboard span grouping is missing.

/arth-setup — add providers, keys, and local runtimes (the installer; eval only validates)
/router — turn graded results into a proposed routing change (scripts/routing-feedback.sh)
/qa — day-to-day quality checks on your current change
Arth Intelligence — where run telemetry and costs are visualized
Workflow comparison — where /eval fits among the toolkit workflows