arthai-marketplace

Measure What AI Costs You

Arth Intelligence records every Claude Code session, tool calls, agent spawns, tokens, dollars, on your own machine. Its Experiments page answers the question this guide is about: what does the toolkit actually change, and what does my AI usage cost? This page is the condensed workflow; the full guide has setup, dashboard tour, and privacy details.

Prerequisite: observability enabled via /otel-setup, see Getting Started Step 7. Cost and token data require Claude Code’s native OTEL (CLAUDE_CODE_ENABLE_TELEMETRY=1, which /otel-setup writes); without it the cost columns stay empty.

Run a baseline vs toolkit A/B

Every session is auto-tagged with an arth.experiment label, no setup before each launch:

Mode Label format
No-toolkit baseline (plain claude, no plugin) auto-baseline-<git-branch>-<first-prompt-slug>-<unix>
Toolkit-on session auto-toolkit-<git-branch>-<unix>

The zero-config flow:

  1. Run the task without the toolkit (plain claude in a project with no plugin).
  2. Run the same task with the toolkit installed.
  3. Open http://localhost:3100/experiments, pick one label on each side, click Compare.

Want telemetry on but toolkit side effects off for the baseline? export OTEL_OBSERVE_ONLY=true before launching. Want a human-readable label instead of the auto one? Set arth.experiment in OTEL_RESOURCE_ATTRIBUTES before launching, your value wins. Auto-tagging can be disabled with ARTH_AUTO_EXPERIMENT_DISABLED=1. Full details: Getting Started 7f.

Reading the six metrics

The comparison fills in side-by-side:

Metric What it tells you
Cost (USD) Total spend per run: the bottom line
Tokens Input/output/cache volume behind that cost
API calls How many model round-trips the run took
Cache hit rate How much context was reused instead of re-paid
Lines edited Output volume of actual code change
Active time Wall-clock engagement, not idle time

Plus workflow attribution, which skills and agents the toolkit-on run used. Two caveats: cost/token/cache metrics need native OTEL (sessions without it show 0), and lines-edited only populates for toolkit-instrumented sessions.

Going deeper

Beyond measuring: route by cost

Measuring tells you what you spent; the Arth Router decides what to spend before the work runs. It routes each task to a model/harness/platform with awareness of the capacity you already paid for, an under-used Max plan window routes to the best models at $0 marginal cost; a burned-down window right-sizes to cheaper ones, governed by budgets and tiers, with every decision explained.

Pairs with this guide’s telemetry: the outcomes bridge feeds actual usage back into the router, so its cost estimates are continuously reconciled against what you really spent.