arthai-marketplace

/eval

Run a model/stack eval study on your own tasks — wizard by default, quick for a zero-decision smoke contrast, deterministic grading, and a report that leads with a plain verdict.

Synopsis

/eval                                  # interactive wizard (the default)
/eval quick                            # canned 3-task set, current stack vs claude-only, one confirm
/eval run --set <set> --arms <csv> [--reps N] [--ceiling USD] [--dry-run]
/eval status                           # cells done, spend vs ceiling, failures
/eval report                           # plain verdict first, tables underneath
/eval run --from-issues owner/repo#12,#34 ...   # scaffold tasks from bug reports

When to use it

You mixed a cheaper model into your stack (via /arth-setup) and want to know whether it silently breaks things while appearing to succeed.
You want a repeatable, versioned comparison — same task set, different arms, across time (including before/after a toolkit release).
You want pass@3 evidence for “capable but inconsistent” vs “incapable” before changing your model policy.
Not for: day-to-day “did my change work?” checks — that’s /qa. Not for adding providers or keys — that’s /arth-setup (eval validates, it never installs).

Quickstart

/eval quick

What you’ll see: a preflight summary, one confirmation showing the matrix (3 tasks × 2 arms) with a cost estimate and a spend ceiling, then a background run. When it finishes, /eval report opens with a one-paragraph verdict like “claude passed 3 of 3 tasks at $0.41 per passing task — deepseek was cheaper but failed more.”

Examples

/eval                                   # wizard: pick set → arms → reps → confirm ceiling
/eval quick                             # zero decisions, one confirm
/eval run --set smoke --arms claude,deepseek --reps 3 --ceiling 10
/eval run --set smoke --arms claude --dry-run     # estimate only, no API calls
/eval status                            # progress of the background run
/eval report                            # verdict, leaderboard, variance strip, drill-down

Arguments & flags

Flag	Values	Default	What it does
`--set`	a set under `tests/eval/tasks/sets/`	—	which named, versioned task set to run
`--arms`	`claude,<provider>[:<model>]`	—	arms to compare; every provider arm must hold a fresh conformance pass
`--reps`	integer	1	repetitions per task (3 gives you pass@3)
`--ceiling`	USD	—	hard spend abort threshold; required for any paid matrix, always the last question
`--dry-run`	—	off	resolve the matrix and estimate cost with zero API calls and zero writes
`--resume`	run id	—	skip cells already completed in the journal
`--from-issues`	`owner/repo#N,#M`	—	scaffold task specs from issue title/body (never from fix diffs)

What it does

Preflight — validates everything, installs nothing. Seven checks in a fixed order: arm legality (fresh probe pass required — an unprobed or failed model is refused with the exact probe command), remote liveness, local readiness, proxy coverage, workspace capacity, telemetry honesty, and the spend gate last. The one exception to “installs nothing”: a missing local model can be pulled, but only after a size-stated confirmation.
Confirmation checkpoint. The resolved matrix, the cost estimate, and the spend ceiling are shown in one final prompt — nothing runs and nothing is spent until you confirm. The equivalent /eval run … one-liner is printed so the study is repeatable.
The run. One frozen checkout per cell (task × arm × rep) at a pinned commit with history truncated — the agent cannot look up the fix. Each cell is a non-interactive session, graded on exit by the task’s own deterministic checks (hidden tests are copied in only after the agent exits). Every cell is journaled to tests/eval/runs.jsonl; interrupting is safe, --resume skips completed cells. Costs are recorded at real descriptor pricing, and the run hard-aborts if spend crosses the ceiling.
Grading. Pass means the task’s checks go green — no AI judging anywhere in the pass rate. Cheat-pattern signals (weakened tests, swallowed exceptions, widened lint ignores) are recorded per cell for you to review, but never change the pass/fail outcome on their own.

Output & artifacts

tests/eval/runs.jsonl — the append-only journal (system of record, machine-local, not committed)
/eval report — plain verdict first, then: leaderboard (pass rate, effective cost per passing task, latency with cloud and local kept separate), a variance strip per task so unstable measurements can’t be over-read, and a per-cell drill-down (gates, cheat markers, resolved endpoint and model)
Scaffolded task specs under tests/eval/tasks/, hidden tests under tests/eval/fixtures/<task-id>/
Runs are tagged arth.experiment=eval:<set>:<task>:<arm>:<rep> so the Arth Intelligence experiments view groups them automatically when telemetry is on

Troubleshooting

Problem	Fix
`no conformance record — refused. Run: scripts/provider-registry.sh probe <name>`	The arm was never probed. Run the printed probe command (or re-run `/arth-setup`), then retry.
`conformance pass is stale … re-probe`	Probe verdicts expire after 14 days. Re-run the printed probe command.
`key <NAME> not present`	Add the key through `/arth-setup` — eval never collects keys itself.
`estimate exceeds the ceiling`	Shrink the matrix (fewer arms/reps) or deliberately raise the ceiling at the confirm step.
Run interrupted mid-matrix	Nothing is lost — re-run with `--resume <run-id>` (shown by `/eval status`); completed cells are skipped.
`telemetry was dark for this run` in the report	The intelligence engine was unreachable during the run. Results are complete (the journal is the system of record); only dashboard span grouping is missing.

/arth-setup — add providers, keys, and local runtimes (the installer; eval only validates)
/router — turn graded results into a proposed routing change (scripts/routing-feedback.sh)
/qa — day-to-day quality checks on your current change
Arth Intelligence — where run telemetry and costs are visualized
Workflow comparison — where /eval fits among the toolkit workflows