Evidence · benchmark v7 · 2026-11-02
What we measured, and how far to trust it
Every number here comes from the aggregate CSVs in the repository, which the figures are drawn from. The raw agent transcripts are not published; a new run uses live advisory data and will differ. The limits sit next to the results.
Method
- Claude Code 2.1.308 in headless mode, driving Sonnet 5.5, Haiku 5.5 and Opus 5.5.
- 12 coding tasks we wrote (8 npm, 4 PyPI) that ask for pinned versions. 2 runs per arm and model.
- Arms: no docs; docs-only (pinright with its verdict off); pinright MCP server; hook only; MCP plus hook. A no-pin reference installs the same packages without a version.
- Each project audited with
npm auditorpip-audit, which share no code with pinright, then installed and built. - 360 agent runs plus 144 docs runs, USD 35.10 in total at API list price, on the 1.0.0 code.
About 30 alerts per project without docs, 0.3 with pinright
Without tools, projects averaged 29.8 to 31.9 whole-tree alerts. With the pinright MCP server every model averaged 0.3, the same as installing the packages with no version. That 0.3 is nuxt's: its current release pulls deepmerge, glob and minimatch versions with advisories and no fix yet.
- Without pinright, verdict off, or no pin
- With pinright
- 95% interval, resampling tasks
40% to 42% of direct pins had a known advisory, 0% with pinright
Sonnet pinned 66 of 164 direct dependencies with an advisory, Haiku 63 of 157, Opus 66 of 159. Examples: axios@1.7.9, pillow@11.0.0, pyjwt@2.10.1. No arm pinned a package or version that does not exist.
- Without pinright, verdict off, or no pin
- With pinright
- 95% interval, resampling tasks
Current docs do most of the work. The verdict does the rest.
The docs-only arm is pinright's own server with the verdict turned off, documenting each package's latest release. Steering the agent to current versions that way cut alerts by 89% to 99%. Without the verdict, Sonnet pinned a vulnerable next@15.5.0 in both of its next-auth runs and Haiku a next-auth beta with advisories in both of its own, and the runs took more turns and cost 1.5x to 1.8x more than with the full server.
| Arm | Sonnet 5.5 | Haiku 5.5 | Opus 5.5 |
|---|---|---|---|
| No docs | 766 (40%) | 715 (40%) | 732 (42%) |
| docs-only (verdict off) | 84 (2%) | 63 (2%) | 6 (0%) |
| pinright MCP | 6 (0%) | 6 (0%) | 6 (0%) |
| pinright hook only | 14 (0%) | 14 (0%) | 14 (0%) |
| pinright MCP + hook | 6 (0%) | 6 (0%) | 6 (0%) |
| Projects that build, pinright MCP | 23/24 | 23/24 | 24/24 |
| Cost per run, no docs / MCP (USD, list price) | 0.033 / 0.050 | 0.003 / 0.006 | 0.074 / 0.126 |
The hook-only remainder comes from next 15.5.28, which has no advisories itself but pulls a postcss with some. On the same 394 lockfiles, pinright's check-manifest flagged the same packages as npm audit and pip-audit (mean Jaccard 0.96 for npm, 1.00 for PyPI).
16 of 16 runs wrote the current API, against 9 to 14 without docs
Eight tasks that only come out right with a library's current major: Next.js 16, Tailwind CSS 4, Zod 4, AI SDK 7, Express 5, ESLint 10, Pydantic 2 and LangChain 1. A run counts when it installs, builds, passes a smoke test, pins the asked major and uses no removed or deprecated API. Without docs, every model wrote AI SDK 5/6 code for AI SDK 7, and most LangChain runs pinned an early 1.x whose imports fail. The models already knew five of the eight APIs, so the gap rests on the other three. The no-docs intervals for Sonnet and Opus overlap 16 of 16; Haiku's does not.
- Without pinright, verdict off, or no pin
- With pinright
- 95% interval, resampling tasks
Without an agent, pinright's docs answered 25 of 28 and 33 of 45 questions in two held-out sets nobody tuned on, over libraries outside the benchmark. A probe checks for strings of the current API, not for a correct answer.
21% to 33% fewer tokens than a docs-only server
Docs MCP servers return documentation for a library; they do not check the version the agent pins, and they do not see the install. The docs-only arm stands in for them: pinright's own server with its security layer off, serving docs for the version asked and no verdict. It is not another product. Against it, the full server used 21% to 26% fewer tokens per run on the agent tasks and 26% to 33% fewer on the docs tasks, and 3.1 to 4.3 fewer turns. No tools at all used the fewest tokens of the three arms, so pinright costs more than nothing.
- No docs, or docs-only context server
- pinright MCP server
- 95% interval, resampling tasks
| Benchmark, model | Tokens (k) | Fewer with pinright | MCP text (k chars) | get-docs calls | Turns | Cost (USD, list price) |
|---|---|---|---|---|---|---|
| Agent, Sonnet 5.5 | 55.6 to 42.2 | 24% [14, 34] | 24.8 to 5.3 | 5.2 to 0.7 | 9.3 to 5.9 | 0.088 to 0.050 |
| Agent, Haiku 5.5 | 93.8 to 74.2 | 21% [6, 33] | 45.3 to 17.2 | 5.7 to 1.7 | 11.1 to 7.5 | 0.008 to 0.005 |
| Agent, Opus 5.5 | 73.3 to 54.6 | 26% [11, 37] | 32.9 to 10.4 | 5.7 to 1.3 | 9.9 to 6.8 | 0.212 to 0.126 |
| Docs, Sonnet 5.5 | 94.0 to 62.9 | 33% [13, 45] | 35.0 to 20.2 | 5.4 to 1.6 | 11.7 to 8.1 | 0.120 to 0.084 |
| Docs, Haiku 5.5 | 135.1 to 95.0 | 30% [4, 46] | 46.9 to 26.1 | 4.5 to 2.0 | 13.4 to 9.1 | 0.010 to 0.007 |
| Docs, Opus 5.5 | 113.2 to 83.9 | 26% [2, 43] | 38.6 to 23.7 | 6.4 to 1.9 | 12.4 to 9.3 | 0.263 to 0.195 |
Why pinright reads less
The transcripts show two ways to find a version. With docs only, the agent used get-docs as its version lookup: in the agent benchmark it called it once per package, before writing any file. 345 of those 398 calls named no version, so they returned the latest release's docs, and 299 put "version" or "latest" in the query. One Sonnet run opens with "I'll check the libraries' current versions in the docs first." With pinright, all 72 agent runs opened with check-packages on the whole dependency list, 0.5k to 0.7k characters per call, which returns the version to pin. The agent then called get-docs 0.7 to 1.7 times per run, with a version in 86 of 87 calls. The full server's instructions ask for that order; the docs-only server has no check-packages, and every prompt asks the agent to check the libraries before choosing versions.
The per-arm token intervals overlap in five of the six model and benchmark pairs; the paired intervals, which compare the same tasks, stay above zero in all six. Cache reads are most of each run's input and cost less than fresh input, so tokens and cost do not move in proportion. Data: tokens_by_model_arm.csv, from bench/tokens.ts.
With a shell, every arm ended at the same 6 alerts
The runs above give the agent Write and Read only, so it pins from memory. We ran the same 12 tasks again with Bash, each job in a disposable Linux container, with a real npm install or pip install and a build check: 216 runs over three models and three arms (no docs, hook only, MCP server and hook). Without the MCP server, the agent ran npm view or pip index versions before pinning in every run. Every arm and model ended with 6 whole-tree alerts, the ones the latest nuxt tree carries, and no direct dependency with an advisory. On these tasks, with a shell, pinright made no measurable difference to what got pinned.
| Arm | Sonnet 5.5 | Opus 5.5 | Haiku 5.5 |
|---|---|---|---|
| No docs | 6 | 6 | 6 |
| Hook only | 6 | 6 | 6 |
| MCP server and hook | 6 | 6 | 6 |
The hook answered 897 tool calls and denied or asked on 48. None stopped a version with an advisory, because no agent tried to install one. 13 were false positives on npm install && npx tsc in one command, and 13 came from a release published hours earlier that deps.dev did not list yet. Hook-only runs took 15% to 21% longer than no docs. Details in RESULTS-shell.md.
The hook blocked or asked on 1.46% of 25,077 public commands
We ran 25,077 commands and manifest writes from 180 public repositories (CI steps, package.json scripts, README lines, Dockerfiles, Makefiles) through the real hook. The 1.0.0 release blocked or asked about 1.46%, and denied 0.13%. Most of those ask on purpose: a version in a variable, a git source, another package index. 32 were true positives, such as npm install -g pnpm@8 with 26 advisories in a public workflow. The 13 root causes we fixed were found on this same corpus, so there is no held-out set. We also replayed a month of our own Claude Code sessions, 154,402 distinct tool calls: after two fixes it prompted, the hook misread 10 of them, against 52 true positives and 79 asks by design.
- Release candidate
- After the fixes this replay prompted
Latency
The hook runs before every shell command. A command that installs nothing took 121 ms at the median; one install took 358 ms with a warm cache and 2.4 s cold. With the MCP server, runs took about 2 to 4 more agent turns and 1.5 to 2 times the API-equivalent cost of a run without tools.
| Case | Cache | p50 | p95 | Budget |
|---|---|---|---|---|
| Command that installs nothing (npm test) | n/a | 121 ms | 177 ms | 238 ms |
| One install (npm i express) | warm | 358 ms | 372 ms | 1 s |
| One install | cold | 2.4 s | 3.7 s | 5 s |
| 15-line requirements.txt | warm | 1.0 s | 1.5 s | 1 s |
| 15-line requirements.txt | cold | 4.6 s | 5.6 s | 5 s |
| package.json with 30 new dependencies | warm | 1.2 s | 1.5 s | 1.5 s |
| package.json with 30 new dependencies | cold | 10.9 s | 15.2 s | 45 s |
The budget for a command that installs nothing is Node's own start (88 ms) plus 150 ms. A cold 15-line requirements file goes over its p95 budget.
Caveats
- One vendor's models, all driven by Claude Code. Other agents may pin differently.
- Tasks we wrote, and new projects only. Editing an old codebase is a different problem.
- The headline runs give the agent no shell. With a shell, the agents looked versions up themselves and every arm ended at the same 6 alerts.
- Advisory data from the two days of the run, 2026-11-02 and 2026-11-03. New advisories change the counts.
- 2 runs per arm and model, so 16 to 24 runs per cell. Read the intervals, not just the bars.
- pinright knows published advisories only. A package with no advisory on record has none reported yet.
- The public false-positive corpus is CI and README commands. The agent-session replay covers one user's month of work.
Data and report
The technical report covers design, method, results and the security reviews. The aggregate CSVs and the scripts that draw every figure from them are in the repository. Benchmark v7 ran on the 1.0.0 code; v6, on a pre-release commit, is kept in the repository for comparison.