Benzi — Benchmarks

Benzi’s KPI (Key Performance Indicator) — source lines read

Every harness opens more source as bugs get harder. The question is the slope.
Each point is one bug; the 24 are laid out easiest to hardest, left to right.

Difficulty is Claude Code’s turn count on that bug — a third-party yardstick, so no
harness sets its own position on the axis. Lines read counts only what came back from
file-read calls; grep and shell output are search, not reading. The figure beside each line is its
slope: how many extra lines that harness opens per step of difficulty. Hover any point for the bug
and its count.

Lines read counts only what came back from file-read calls; grep and shell output are search, not reading. The green figure in each row is the lowest of the four.

The same 24 bugs in the same order, with wall clock in place of lines read.

Wall clock is raw here — unlike the tables above, Benzi’s per-repo index build is
not subtracted, so these seconds run slightly higher than the warm figures quoted elsewhere on this page.
Each point is that harness’s most recent solved run for that bug; unsolved and unfinished runs are left
out rather than plotted as fast. Benzi on Sonnet never solved http-parser and the DeepSeek harness never
ran nats-server, so those two points are absent and neither enters its fit.

And the same again with dollars on the vertical axis.

Priced at the published per-token rates, same run selection as the chart above it. The
two DeepSeek series run an order of magnitude cheaper than the two Sonnet ones, so at this scale they
sit close to the baseline — the per-bug figures behind them are in the DeepSeek table further down.
What the axis does show is the slope: Claude Code’s cost climbs with difficulty faster than any
other series here.

Priced at published per-token rates. The green figure in each row is the lowest of the four; the two DeepSeek columns are cheaper largely because that model costs roughly twenty times less per token. Blank cells are the two runs that never produced a fix.