Evidence · benchmark v7 · 2026-11-02

What we measured, and how far to trust it

Every number here comes from the aggregate CSVs in the repository, which the figures are drawn from. The raw agent transcripts are not published; a new run uses live advisory data and will differ. The limits sit next to the results.

Method

  • Claude Code 2.1.308 in headless mode, driving Sonnet 5.5, Haiku 5.5 and Opus 5.5.
  • 12 coding tasks we wrote (8 npm, 4 PyPI) that ask for pinned versions. 2 runs per arm and model.
  • Arms: no docs; docs-only (pinright with its verdict off); pinright MCP server; hook only; MCP plus hook. A no-pin reference installs the same packages without a version.
  • Each project audited with npm audit or pip-audit, which share no code with pinright, then installed and built.
  • 360 agent runs plus 144 docs runs, USD 35.10 in total at API list price, on the 1.0.0 code.

About 30 alerts per project without docs, 0.3 with pinright

Without tools, projects averaged 29.8 to 31.9 whole-tree alerts. With the pinright MCP server every model averaged 0.3, the same as installing the packages with no version. That 0.3 is nuxt's: its current release pulls deepmerge, glob and minimatch versions with advisories and no fix yet.

  • Without pinright, verdict off, or no pin
  • With pinright
  • 95% interval, resampling tasks
Whole-tree alerts per project, Sonnet 5.5Mean whole-tree alerts per project for every arm with 95% intervals. Sonnet 5.5, No docs: 31.9 whole-tree alerts per project (95% interval 20.3 to 44.7, 24 projects). Sonnet 5.5, Docs only: 3.5 whole-tree alerts per project (95% interval 0.0 to 10.0, 24 projects). Sonnet 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Sonnet 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.4, 24 projects). Sonnet 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 23 projects). Sonnet 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects). Opus 5.5, No docs: 30.5 whole-tree alerts per project (95% interval 18.0 to 43.5, 24 projects). Opus 5.5, Docs only: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Opus 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Opus 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.4, 24 projects). Opus 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Opus 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects). Haiku 5.5, No docs: 29.8 whole-tree alerts per project (95% interval 17.8 to 43.5, 24 projects). Haiku 5.5, Docs only: 2.6 whole-tree alerts per project (95% interval 0.0 to 7.6, 24 projects). Haiku 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Haiku 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.5, 23 projects). Haiku 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Haiku 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects)01020304050Sonnet 5.5Sonnet 5.5, No docs: 31.9 whole-tree alerts per project (95% interval 20.3 to 44.7, 24 projects)No docs31.9Sonnet 5.5, Docs only: 3.5 whole-tree alerts per project (95% interval 0.0 to 10.0, 24 projects)Docs only3.5Sonnet 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects)pinright MCP0.3Sonnet 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.4, 24 projects)Hook only0.6Sonnet 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 23 projects)MCP + hook0.3Sonnet 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects)No pin0.3
Whole-tree alerts per project, Opus 5.5Mean whole-tree alerts per project for every arm with 95% intervals. Sonnet 5.5, No docs: 31.9 whole-tree alerts per project (95% interval 20.3 to 44.7, 24 projects). Sonnet 5.5, Docs only: 3.5 whole-tree alerts per project (95% interval 0.0 to 10.0, 24 projects). Sonnet 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Sonnet 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.4, 24 projects). Sonnet 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 23 projects). Sonnet 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects). Opus 5.5, No docs: 30.5 whole-tree alerts per project (95% interval 18.0 to 43.5, 24 projects). Opus 5.5, Docs only: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Opus 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Opus 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.4, 24 projects). Opus 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Opus 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects). Haiku 5.5, No docs: 29.8 whole-tree alerts per project (95% interval 17.8 to 43.5, 24 projects). Haiku 5.5, Docs only: 2.6 whole-tree alerts per project (95% interval 0.0 to 7.6, 24 projects). Haiku 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Haiku 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.5, 23 projects). Haiku 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Haiku 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects)01020304050Opus 5.5Opus 5.5, No docs: 30.5 whole-tree alerts per project (95% interval 18.0 to 43.5, 24 projects)No docs30.5Opus 5.5, Docs only: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects)Docs only0.3Opus 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects)pinright MCP0.3Opus 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.4, 24 projects)Hook only0.6Opus 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects)MCP + hook0.3Opus 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects)No pin0.3
Whole-tree alerts per project, Haiku 5.5Mean whole-tree alerts per project for every arm with 95% intervals. Sonnet 5.5, No docs: 31.9 whole-tree alerts per project (95% interval 20.3 to 44.7, 24 projects). Sonnet 5.5, Docs only: 3.5 whole-tree alerts per project (95% interval 0.0 to 10.0, 24 projects). Sonnet 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Sonnet 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.4, 24 projects). Sonnet 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 23 projects). Sonnet 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects). Opus 5.5, No docs: 30.5 whole-tree alerts per project (95% interval 18.0 to 43.5, 24 projects). Opus 5.5, Docs only: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Opus 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Opus 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.4, 24 projects). Opus 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Opus 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects). Haiku 5.5, No docs: 29.8 whole-tree alerts per project (95% interval 17.8 to 43.5, 24 projects). Haiku 5.5, Docs only: 2.6 whole-tree alerts per project (95% interval 0.0 to 7.6, 24 projects). Haiku 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Haiku 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.5, 23 projects). Haiku 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects). Haiku 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects)01020304050Haiku 5.5Haiku 5.5, No docs: 29.8 whole-tree alerts per project (95% interval 17.8 to 43.5, 24 projects)No docs29.8Haiku 5.5, Docs only: 2.6 whole-tree alerts per project (95% interval 0.0 to 7.6, 24 projects)Docs only2.6Haiku 5.5, pinright MCP: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects)pinright MCP0.3Haiku 5.5, Hook only: 0.6 whole-tree alerts per project (95% interval 0.0 to 1.5, 23 projects)Hook only0.6Haiku 5.5, MCP + hook: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 24 projects)MCP + hook0.3Haiku 5.5, No pin: 0.3 whole-tree alerts per project (95% interval 0.0 to 0.8, 12 projects)No pin0.3
Whole-tree alerts per project. Mean npm audit / pip-audit alerts per generated project, 12 tasks x 2 runs per arm and model (No pin: 1 run per task). Whiskers: 95% cluster-bootstrap intervals that resample tasks. Same scale in the three panels.

40% to 42% of direct pins had a known advisory, 0% with pinright

Sonnet pinned 66 of 164 direct dependencies with an advisory, Haiku 63 of 157, Opus 66 of 159. Examples: axios@1.7.9, pillow@11.0.0, pyjwt@2.10.1. No arm pinned a package or version that does not exist.

  • Without pinright, verdict off, or no pin
  • With pinright
  • 95% interval, resampling tasks
Direct dependencies with a known advisory, Sonnet 5.5Share of pinned direct dependencies with a known advisory for every arm with 95% intervals. Sonnet 5.5, No docs: 66 of 164 direct dependencies with a known advisory (40%, 95% interval 29% to 50%). Sonnet 5.5, Docs only: 2 of 125 direct dependencies with a known advisory (2%, 95% interval 0% to 6%). Sonnet 5.5, pinright MCP: 0 of 131 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Sonnet 5.5, Hook only: 0 of 160 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Sonnet 5.5, MCP + hook: 0 of 122 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Sonnet 5.5, No pin: 0 of 83 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, No docs: 66 of 159 direct dependencies with a known advisory (42%, 95% interval 32% to 50%). Opus 5.5, Docs only: 0 of 126 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, pinright MCP: 0 of 154 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, Hook only: 0 of 156 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, MCP + hook: 0 of 149 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, No pin: 0 of 78 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, No docs: 63 of 157 direct dependencies with a known advisory (40%, 95% interval 30% to 49%). Haiku 5.5, Docs only: 3 of 142 direct dependencies with a known advisory (2%, 95% interval 0% to 7%). Haiku 5.5, pinright MCP: 0 of 137 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, Hook only: 0 of 148 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, MCP + hook: 0 of 123 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, No pin: 0 of 76 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)0%20%40%Sonnet 5.5Sonnet 5.5, No docs: 66 of 164 direct dependencies with a known advisory (40%, 95% interval 29% to 50%)No docs40%Sonnet 5.5, Docs only: 2 of 125 direct dependencies with a known advisory (2%, 95% interval 0% to 6%)Docs only2%Sonnet 5.5, pinright MCP: 0 of 131 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)pinright MCP0%Sonnet 5.5, Hook only: 0 of 160 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)Hook only0%Sonnet 5.5, MCP + hook: 0 of 122 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)MCP + hook0%Sonnet 5.5, No pin: 0 of 83 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)No pin0%
Direct dependencies with a known advisory, Opus 5.5Share of pinned direct dependencies with a known advisory for every arm with 95% intervals. Sonnet 5.5, No docs: 66 of 164 direct dependencies with a known advisory (40%, 95% interval 29% to 50%). Sonnet 5.5, Docs only: 2 of 125 direct dependencies with a known advisory (2%, 95% interval 0% to 6%). Sonnet 5.5, pinright MCP: 0 of 131 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Sonnet 5.5, Hook only: 0 of 160 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Sonnet 5.5, MCP + hook: 0 of 122 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Sonnet 5.5, No pin: 0 of 83 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, No docs: 66 of 159 direct dependencies with a known advisory (42%, 95% interval 32% to 50%). Opus 5.5, Docs only: 0 of 126 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, pinright MCP: 0 of 154 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, Hook only: 0 of 156 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, MCP + hook: 0 of 149 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, No pin: 0 of 78 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, No docs: 63 of 157 direct dependencies with a known advisory (40%, 95% interval 30% to 49%). Haiku 5.5, Docs only: 3 of 142 direct dependencies with a known advisory (2%, 95% interval 0% to 7%). Haiku 5.5, pinright MCP: 0 of 137 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, Hook only: 0 of 148 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, MCP + hook: 0 of 123 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, No pin: 0 of 76 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)0%20%40%Opus 5.5Opus 5.5, No docs: 66 of 159 direct dependencies with a known advisory (42%, 95% interval 32% to 50%)No docs42%Opus 5.5, Docs only: 0 of 126 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)Docs only0%Opus 5.5, pinright MCP: 0 of 154 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)pinright MCP0%Opus 5.5, Hook only: 0 of 156 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)Hook only0%Opus 5.5, MCP + hook: 0 of 149 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)MCP + hook0%Opus 5.5, No pin: 0 of 78 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)No pin0%
Direct dependencies with a known advisory, Haiku 5.5Share of pinned direct dependencies with a known advisory for every arm with 95% intervals. Sonnet 5.5, No docs: 66 of 164 direct dependencies with a known advisory (40%, 95% interval 29% to 50%). Sonnet 5.5, Docs only: 2 of 125 direct dependencies with a known advisory (2%, 95% interval 0% to 6%). Sonnet 5.5, pinright MCP: 0 of 131 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Sonnet 5.5, Hook only: 0 of 160 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Sonnet 5.5, MCP + hook: 0 of 122 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Sonnet 5.5, No pin: 0 of 83 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, No docs: 66 of 159 direct dependencies with a known advisory (42%, 95% interval 32% to 50%). Opus 5.5, Docs only: 0 of 126 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, pinright MCP: 0 of 154 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, Hook only: 0 of 156 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, MCP + hook: 0 of 149 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Opus 5.5, No pin: 0 of 78 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, No docs: 63 of 157 direct dependencies with a known advisory (40%, 95% interval 30% to 49%). Haiku 5.5, Docs only: 3 of 142 direct dependencies with a known advisory (2%, 95% interval 0% to 7%). Haiku 5.5, pinright MCP: 0 of 137 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, Hook only: 0 of 148 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, MCP + hook: 0 of 123 direct dependencies with a known advisory (0%, 95% interval 0% to 0%). Haiku 5.5, No pin: 0 of 76 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)0%20%40%Haiku 5.5Haiku 5.5, No docs: 63 of 157 direct dependencies with a known advisory (40%, 95% interval 30% to 49%)No docs40%Haiku 5.5, Docs only: 3 of 142 direct dependencies with a known advisory (2%, 95% interval 0% to 7%)Docs only2%Haiku 5.5, pinright MCP: 0 of 137 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)pinright MCP0%Haiku 5.5, Hook only: 0 of 148 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)Hook only0%Haiku 5.5, MCP + hook: 0 of 123 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)MCP + hook0%Haiku 5.5, No pin: 0 of 76 direct dependencies with a known advisory (0%, 95% interval 0% to 0%)No pin0%
Direct dependencies pinned with a known advisory. At least one deps.dev advisory when the runs were scored (2026-11-02 and 2026-11-03), with 95% cluster-bootstrap intervals. deps.dev is also the source pinright reads, so the 0% in its arms follows from how it picks versions. The whole-tree alerts above come from npm audit and pip-audit, which do not share that path.

Current docs do most of the work. The verdict does the rest.

The docs-only arm is pinright's own server with the verdict turned off, documenting each package's latest release. Steering the agent to current versions that way cut alerts by 89% to 99%. Without the verdict, Sonnet pinned a vulnerable next@15.5.0 in both of its next-auth runs and Haiku a next-auth beta with advisories in both of its own, and the runs took more turns and cost 1.5x to 1.8x more than with the full server.

Whole-tree alerts, sum over 24 projects (direct deps with an advisory)
ArmSonnet 5.5Haiku 5.5Opus 5.5
No docs766 (40%)715 (40%)732 (42%)
docs-only (verdict off)84 (2%)63 (2%)6 (0%)
pinright MCP6 (0%)6 (0%)6 (0%)
pinright hook only14 (0%)14 (0%)14 (0%)
pinright MCP + hook6 (0%)6 (0%)6 (0%)
Projects that build, pinright MCP23/2423/2424/24
Cost per run, no docs / MCP (USD, list price)0.033 / 0.0500.003 / 0.0060.074 / 0.126

The hook-only remainder comes from next 15.5.28, which has no advisories itself but pulls a postcss with some. On the same 394 lockfiles, pinright's check-manifest flagged the same packages as npm audit and pip-audit (mean Jaccard 0.96 for npm, 1.00 for PyPI).

16 of 16 runs wrote the current API, against 9 to 14 without docs

Eight tasks that only come out right with a library's current major: Next.js 16, Tailwind CSS 4, Zod 4, AI SDK 7, Express 5, ESLint 10, Pydantic 2 and LangChain 1. A run counts when it installs, builds, passes a smoke test, pins the asked major and uses no removed or deprecated API. Without docs, every model wrote AI SDK 5/6 code for AI SDK 7, and most LangChain runs pinned an early 1.x whose imports fail. The models already knew five of the eight APIs, so the gap rests on the other three. The no-docs intervals for Sonnet and Opus overlap 16 of 16; Haiku's does not.

  • Without pinright, verdict off, or no pin
  • With pinright
  • 95% interval, resampling tasks
Docs runs that wrote the current API, Sonnet 5.5Docs benchmark runs out of 16 that wrote the current API, with 95% intervals. Sonnet 5.5, No docs: 13 of 16 runs wrote the current API. Sonnet 5.5, Docs only: 16 of 16 runs wrote the current API. Sonnet 5.5, pinright MCP: 16 of 16 runs wrote the current API. Opus 5.5, No docs: 14 of 16 runs wrote the current API. Opus 5.5, Docs only: 16 of 16 runs wrote the current API. Opus 5.5, pinright MCP: 16 of 16 runs wrote the current API. Haiku 5.5, No docs: 9 of 16 runs wrote the current API. Haiku 5.5, Docs only: 15 of 16 runs wrote the current API. Haiku 5.5, pinright MCP: 16 of 16 runs wrote the current API0481216Sonnet 5.5Sonnet 5.5, No docs: 13 of 16 runs wrote the current APINo docs13 of 16Sonnet 5.5, Docs only: 16 of 16 runs wrote the current APIDocs only16 of 16Sonnet 5.5, pinright MCP: 16 of 16 runs wrote the current APIpinright MCP16 of 16
Docs runs that wrote the current API, Opus 5.5Docs benchmark runs out of 16 that wrote the current API, with 95% intervals. Sonnet 5.5, No docs: 13 of 16 runs wrote the current API. Sonnet 5.5, Docs only: 16 of 16 runs wrote the current API. Sonnet 5.5, pinright MCP: 16 of 16 runs wrote the current API. Opus 5.5, No docs: 14 of 16 runs wrote the current API. Opus 5.5, Docs only: 16 of 16 runs wrote the current API. Opus 5.5, pinright MCP: 16 of 16 runs wrote the current API. Haiku 5.5, No docs: 9 of 16 runs wrote the current API. Haiku 5.5, Docs only: 15 of 16 runs wrote the current API. Haiku 5.5, pinright MCP: 16 of 16 runs wrote the current API0481216Opus 5.5Opus 5.5, No docs: 14 of 16 runs wrote the current APINo docs14 of 16Opus 5.5, Docs only: 16 of 16 runs wrote the current APIDocs only16 of 16Opus 5.5, pinright MCP: 16 of 16 runs wrote the current APIpinright MCP16 of 16
Docs runs that wrote the current API, Haiku 5.5Docs benchmark runs out of 16 that wrote the current API, with 95% intervals. Sonnet 5.5, No docs: 13 of 16 runs wrote the current API. Sonnet 5.5, Docs only: 16 of 16 runs wrote the current API. Sonnet 5.5, pinright MCP: 16 of 16 runs wrote the current API. Opus 5.5, No docs: 14 of 16 runs wrote the current API. Opus 5.5, Docs only: 16 of 16 runs wrote the current API. Opus 5.5, pinright MCP: 16 of 16 runs wrote the current API. Haiku 5.5, No docs: 9 of 16 runs wrote the current API. Haiku 5.5, Docs only: 15 of 16 runs wrote the current API. Haiku 5.5, pinright MCP: 16 of 16 runs wrote the current API0481216Haiku 5.5Haiku 5.5, No docs: 9 of 16 runs wrote the current APINo docs9 of 16Haiku 5.5, Docs only: 15 of 16 runs wrote the current APIDocs only15 of 16Haiku 5.5, pinright MCP: 16 of 16 runs wrote the current APIpinright MCP16 of 16
Docs benchmark. 16 runs per arm and model; whiskers are 95% intervals. "16 of 16" is a measured result on these tasks, not an accuracy rate.

Without an agent, pinright's docs answered 25 of 28 and 33 of 45 questions in two held-out sets nobody tuned on, over libraries outside the benchmark. A probe checks for strings of the current API, not for a correct answer.

21% to 33% fewer tokens than a docs-only server

Docs MCP servers return documentation for a library; they do not check the version the agent pins, and they do not see the install. The docs-only arm stands in for them: pinright's own server with its security layer off, serving docs for the version asked and no verdict. It is not another product. Against it, the full server used 21% to 26% fewer tokens per run on the agent tasks and 26% to 33% fewer on the docs tasks, and 3.1 to 4.3 fewer turns. No tools at all used the fewest tokens of the three arms, so pinright costs more than nothing.

  • No docs, or docs-only context server
  • pinright MCP server
  • 95% interval, resampling tasks
Tokens per run, agent tasks, Sonnet 5.5Mean tokens per run on the 12 agent tasks for no docs, docs only and the pinright MCP server, with 95% intervals. Sonnet 5.5, No docs, agent tasks: 15.3k tokens per run (95% interval 14.6k to 16.1k, 24 runs). Sonnet 5.5, Docs only, agent tasks: 55.6k tokens per run (95% interval 47.2k to 63.5k, 24 runs). Sonnet 5.5, pinright MCP, agent tasks: 42.2k tokens per run (95% interval 37.4k to 46.8k, 24 runs); 24% less than docs only on the same tasks. Opus 5.5, No docs, agent tasks: 16.4k tokens per run (95% interval 15.3k to 18.0k, 24 runs). Opus 5.5, Docs only, agent tasks: 73.3k tokens per run (95% interval 58.8k to 88.5k, 24 runs). Opus 5.5, pinright MCP, agent tasks: 54.6k tokens per run (95% interval 45.2k to 65.0k, 24 runs); 26% less than docs only on the same tasks. Haiku 5.5, No docs, agent tasks: 20.8k tokens per run (95% interval 19.8k to 22.0k, 24 runs). Haiku 5.5, Docs only, agent tasks: 93.8k tokens per run (95% interval 73.5k to 114.8k, 24 runs). Haiku 5.5, pinright MCP, agent tasks: 74.2k tokens per run (95% interval 61.8k to 87.8k, 24 runs); 21% less than docs only on the same tasks050k100k150kSonnet 5.5Sonnet 5.5, No docs, agent tasks: 15.3k tokens per run (95% interval 14.6k to 16.1k, 24 runs)No docs15.3kSonnet 5.5, Docs only, agent tasks: 55.6k tokens per run (95% interval 47.2k to 63.5k, 24 runs)Docs only55.6kSonnet 5.5, pinright MCP, agent tasks: 42.2k tokens per run (95% interval 37.4k to 46.8k, 24 runs); 24% less than docs only on the same taskspinright MCP42.2k
Tokens per run, agent tasks, Opus 5.5Mean tokens per run on the 12 agent tasks for no docs, docs only and the pinright MCP server, with 95% intervals. Sonnet 5.5, No docs, agent tasks: 15.3k tokens per run (95% interval 14.6k to 16.1k, 24 runs). Sonnet 5.5, Docs only, agent tasks: 55.6k tokens per run (95% interval 47.2k to 63.5k, 24 runs). Sonnet 5.5, pinright MCP, agent tasks: 42.2k tokens per run (95% interval 37.4k to 46.8k, 24 runs); 24% less than docs only on the same tasks. Opus 5.5, No docs, agent tasks: 16.4k tokens per run (95% interval 15.3k to 18.0k, 24 runs). Opus 5.5, Docs only, agent tasks: 73.3k tokens per run (95% interval 58.8k to 88.5k, 24 runs). Opus 5.5, pinright MCP, agent tasks: 54.6k tokens per run (95% interval 45.2k to 65.0k, 24 runs); 26% less than docs only on the same tasks. Haiku 5.5, No docs, agent tasks: 20.8k tokens per run (95% interval 19.8k to 22.0k, 24 runs). Haiku 5.5, Docs only, agent tasks: 93.8k tokens per run (95% interval 73.5k to 114.8k, 24 runs). Haiku 5.5, pinright MCP, agent tasks: 74.2k tokens per run (95% interval 61.8k to 87.8k, 24 runs); 21% less than docs only on the same tasks050k100k150kOpus 5.5Opus 5.5, No docs, agent tasks: 16.4k tokens per run (95% interval 15.3k to 18.0k, 24 runs)No docs16.4kOpus 5.5, Docs only, agent tasks: 73.3k tokens per run (95% interval 58.8k to 88.5k, 24 runs)Docs only73.3kOpus 5.5, pinright MCP, agent tasks: 54.6k tokens per run (95% interval 45.2k to 65.0k, 24 runs); 26% less than docs only on the same taskspinright MCP54.6k
Tokens per run, agent tasks, Haiku 5.5Mean tokens per run on the 12 agent tasks for no docs, docs only and the pinright MCP server, with 95% intervals. Sonnet 5.5, No docs, agent tasks: 15.3k tokens per run (95% interval 14.6k to 16.1k, 24 runs). Sonnet 5.5, Docs only, agent tasks: 55.6k tokens per run (95% interval 47.2k to 63.5k, 24 runs). Sonnet 5.5, pinright MCP, agent tasks: 42.2k tokens per run (95% interval 37.4k to 46.8k, 24 runs); 24% less than docs only on the same tasks. Opus 5.5, No docs, agent tasks: 16.4k tokens per run (95% interval 15.3k to 18.0k, 24 runs). Opus 5.5, Docs only, agent tasks: 73.3k tokens per run (95% interval 58.8k to 88.5k, 24 runs). Opus 5.5, pinright MCP, agent tasks: 54.6k tokens per run (95% interval 45.2k to 65.0k, 24 runs); 26% less than docs only on the same tasks. Haiku 5.5, No docs, agent tasks: 20.8k tokens per run (95% interval 19.8k to 22.0k, 24 runs). Haiku 5.5, Docs only, agent tasks: 93.8k tokens per run (95% interval 73.5k to 114.8k, 24 runs). Haiku 5.5, pinright MCP, agent tasks: 74.2k tokens per run (95% interval 61.8k to 87.8k, 24 runs); 21% less than docs only on the same tasks050k100k150kHaiku 5.5Haiku 5.5, No docs, agent tasks: 20.8k tokens per run (95% interval 19.8k to 22.0k, 24 runs)No docs20.8kHaiku 5.5, Docs only, agent tasks: 93.8k tokens per run (95% interval 73.5k to 114.8k, 24 runs)Docs only93.8kHaiku 5.5, pinright MCP, agent tasks: 74.2k tokens per run (95% interval 61.8k to 87.8k, 24 runs); 21% less than docs only on the same taskspinright MCP74.2k
Tokens per run, agent benchmark. Everything the model processed in a run (uncached input, cache reads and writes, output), from the usage record Claude Code prices the run from. 12 tasks x 2 runs per arm and model. Same scale in the three panels and in the docs chart below.
Tokens per run, docs tasks, Sonnet 5.5Mean tokens per run on the 8 docs tasks for no docs, docs only and the pinright MCP server, with 95% intervals. Sonnet 5.5, No docs, docs tasks: 17.2k tokens per run (95% interval 16.3k to 18.2k, 16 runs). Sonnet 5.5, Docs only, docs tasks: 94.0k tokens per run (95% interval 63.8k to 125.1k, 16 runs). Sonnet 5.5, pinright MCP, docs tasks: 62.9k tokens per run (95% interval 52.4k to 76.7k, 16 runs); 33% less than docs only on the same tasks. Opus 5.5, No docs, docs tasks: 19.0k tokens per run (95% interval 17.3k to 21.0k, 16 runs). Opus 5.5, Docs only, docs tasks: 113.2k tokens per run (95% interval 78.8k to 152.6k, 16 runs). Opus 5.5, pinright MCP, docs tasks: 83.9k tokens per run (95% interval 67.1k to 98.9k, 16 runs); 26% less than docs only on the same tasks. Haiku 5.5, No docs, docs tasks: 23.5k tokens per run (95% interval 21.2k to 26.2k, 16 runs). Haiku 5.5, Docs only, docs tasks: 135.1k tokens per run (95% interval 95.7k to 179.9k, 16 runs). Haiku 5.5, pinright MCP, docs tasks: 95.0k tokens per run (95% interval 77.2k to 113.3k, 16 runs); 30% less than docs only on the same tasks050k100k150kSonnet 5.5Sonnet 5.5, No docs, docs tasks: 17.2k tokens per run (95% interval 16.3k to 18.2k, 16 runs)No docs17.2kSonnet 5.5, Docs only, docs tasks: 94.0k tokens per run (95% interval 63.8k to 125.1k, 16 runs)Docs only94.0kSonnet 5.5, pinright MCP, docs tasks: 62.9k tokens per run (95% interval 52.4k to 76.7k, 16 runs); 33% less than docs only on the same taskspinright MCP62.9k
Tokens per run, docs tasks, Opus 5.5Mean tokens per run on the 8 docs tasks for no docs, docs only and the pinright MCP server, with 95% intervals. Sonnet 5.5, No docs, docs tasks: 17.2k tokens per run (95% interval 16.3k to 18.2k, 16 runs). Sonnet 5.5, Docs only, docs tasks: 94.0k tokens per run (95% interval 63.8k to 125.1k, 16 runs). Sonnet 5.5, pinright MCP, docs tasks: 62.9k tokens per run (95% interval 52.4k to 76.7k, 16 runs); 33% less than docs only on the same tasks. Opus 5.5, No docs, docs tasks: 19.0k tokens per run (95% interval 17.3k to 21.0k, 16 runs). Opus 5.5, Docs only, docs tasks: 113.2k tokens per run (95% interval 78.8k to 152.6k, 16 runs). Opus 5.5, pinright MCP, docs tasks: 83.9k tokens per run (95% interval 67.1k to 98.9k, 16 runs); 26% less than docs only on the same tasks. Haiku 5.5, No docs, docs tasks: 23.5k tokens per run (95% interval 21.2k to 26.2k, 16 runs). Haiku 5.5, Docs only, docs tasks: 135.1k tokens per run (95% interval 95.7k to 179.9k, 16 runs). Haiku 5.5, pinright MCP, docs tasks: 95.0k tokens per run (95% interval 77.2k to 113.3k, 16 runs); 30% less than docs only on the same tasks050k100k150kOpus 5.5Opus 5.5, No docs, docs tasks: 19.0k tokens per run (95% interval 17.3k to 21.0k, 16 runs)No docs19.0kOpus 5.5, Docs only, docs tasks: 113.2k tokens per run (95% interval 78.8k to 152.6k, 16 runs)Docs only113.2kOpus 5.5, pinright MCP, docs tasks: 83.9k tokens per run (95% interval 67.1k to 98.9k, 16 runs); 26% less than docs only on the same taskspinright MCP83.9k
Tokens per run, docs tasks, Haiku 5.5Mean tokens per run on the 8 docs tasks for no docs, docs only and the pinright MCP server, with 95% intervals. Sonnet 5.5, No docs, docs tasks: 17.2k tokens per run (95% interval 16.3k to 18.2k, 16 runs). Sonnet 5.5, Docs only, docs tasks: 94.0k tokens per run (95% interval 63.8k to 125.1k, 16 runs). Sonnet 5.5, pinright MCP, docs tasks: 62.9k tokens per run (95% interval 52.4k to 76.7k, 16 runs); 33% less than docs only on the same tasks. Opus 5.5, No docs, docs tasks: 19.0k tokens per run (95% interval 17.3k to 21.0k, 16 runs). Opus 5.5, Docs only, docs tasks: 113.2k tokens per run (95% interval 78.8k to 152.6k, 16 runs). Opus 5.5, pinright MCP, docs tasks: 83.9k tokens per run (95% interval 67.1k to 98.9k, 16 runs); 26% less than docs only on the same tasks. Haiku 5.5, No docs, docs tasks: 23.5k tokens per run (95% interval 21.2k to 26.2k, 16 runs). Haiku 5.5, Docs only, docs tasks: 135.1k tokens per run (95% interval 95.7k to 179.9k, 16 runs). Haiku 5.5, pinright MCP, docs tasks: 95.0k tokens per run (95% interval 77.2k to 113.3k, 16 runs); 30% less than docs only on the same tasks050k100k150kHaiku 5.5Haiku 5.5, No docs, docs tasks: 23.5k tokens per run (95% interval 21.2k to 26.2k, 16 runs)No docs23.5kHaiku 5.5, Docs only, docs tasks: 135.1k tokens per run (95% interval 95.7k to 179.9k, 16 runs)Docs only135.1kHaiku 5.5, pinright MCP, docs tasks: 95.0k tokens per run (95% interval 77.2k to 113.3k, 16 runs); 30% less than docs only on the same taskspinright MCP95.0k
Tokens per run, docs benchmark. 8 tasks x 2 runs per arm and model.
Docs-only context server to pinright, means per run. "Fewer" pairs the two arms task by task; brackets are 95% intervals that resample tasks.
Benchmark, modelTokens (k)Fewer with pinrightMCP text (k chars)get-docs callsTurnsCost (USD, list price)
Agent, Sonnet 5.555.6 to 42.224% [14, 34]24.8 to 5.35.2 to 0.79.3 to 5.90.088 to 0.050
Agent, Haiku 5.593.8 to 74.221% [6, 33]45.3 to 17.25.7 to 1.711.1 to 7.50.008 to 0.005
Agent, Opus 5.573.3 to 54.626% [11, 37]32.9 to 10.45.7 to 1.39.9 to 6.80.212 to 0.126
Docs, Sonnet 5.594.0 to 62.933% [13, 45]35.0 to 20.25.4 to 1.611.7 to 8.10.120 to 0.084
Docs, Haiku 5.5135.1 to 95.030% [4, 46]46.9 to 26.14.5 to 2.013.4 to 9.10.010 to 0.007
Docs, Opus 5.5113.2 to 83.926% [2, 43]38.6 to 23.76.4 to 1.912.4 to 9.30.263 to 0.195

Why pinright reads less

The transcripts show two ways to find a version. With docs only, the agent used get-docs as its version lookup: in the agent benchmark it called it once per package, before writing any file. 345 of those 398 calls named no version, so they returned the latest release's docs, and 299 put "version" or "latest" in the query. One Sonnet run opens with "I'll check the libraries' current versions in the docs first." With pinright, all 72 agent runs opened with check-packages on the whole dependency list, 0.5k to 0.7k characters per call, which returns the version to pin. The agent then called get-docs 0.7 to 1.7 times per run, with a version in 86 of 87 calls. The full server's instructions ask for that order; the docs-only server has no check-packages, and every prompt asks the agent to check the libraries before choosing versions.

MCP tool text per run, agent tasks, Sonnet 5.5Thousands of characters the MCP tools returned per run on the 12 agent tasks, docs only against the pinright MCP server, with 95% intervals. Sonnet 5.5, Docs only, agent tasks: 24.8k characters of MCP tool text per run (95% interval 19.6k to 30.8k, 24 runs). Sonnet 5.5, pinright MCP, agent tasks: 5.3k characters of MCP tool text per run (95% interval 2.9k to 7.7k, 24 runs); 79% less than docs only on the same tasks. Opus 5.5, Docs only, agent tasks: 32.9k characters of MCP tool text per run (95% interval 25.7k to 40.5k, 24 runs). Opus 5.5, pinright MCP, agent tasks: 10.4k characters of MCP tool text per run (95% interval 6.5k to 14.6k, 24 runs); 68% less than docs only on the same tasks. Haiku 5.5, Docs only, agent tasks: 45.3k characters of MCP tool text per run (95% interval 33.3k to 60.6k, 24 runs). Haiku 5.5, pinright MCP, agent tasks: 17.2k characters of MCP tool text per run (95% interval 12.4k to 21.9k, 24 runs); 62% less than docs only on the same tasks020k40k60kSonnet 5.5Sonnet 5.5, Docs only, agent tasks: 24.8k characters of MCP tool text per run (95% interval 19.6k to 30.8k, 24 runs)Docs only24.8kSonnet 5.5, pinright MCP, agent tasks: 5.3k characters of MCP tool text per run (95% interval 2.9k to 7.7k, 24 runs); 79% less than docs only on the same taskspinright MCP5.3k
MCP tool text per run, agent tasks, Opus 5.5Thousands of characters the MCP tools returned per run on the 12 agent tasks, docs only against the pinright MCP server, with 95% intervals. Sonnet 5.5, Docs only, agent tasks: 24.8k characters of MCP tool text per run (95% interval 19.6k to 30.8k, 24 runs). Sonnet 5.5, pinright MCP, agent tasks: 5.3k characters of MCP tool text per run (95% interval 2.9k to 7.7k, 24 runs); 79% less than docs only on the same tasks. Opus 5.5, Docs only, agent tasks: 32.9k characters of MCP tool text per run (95% interval 25.7k to 40.5k, 24 runs). Opus 5.5, pinright MCP, agent tasks: 10.4k characters of MCP tool text per run (95% interval 6.5k to 14.6k, 24 runs); 68% less than docs only on the same tasks. Haiku 5.5, Docs only, agent tasks: 45.3k characters of MCP tool text per run (95% interval 33.3k to 60.6k, 24 runs). Haiku 5.5, pinright MCP, agent tasks: 17.2k characters of MCP tool text per run (95% interval 12.4k to 21.9k, 24 runs); 62% less than docs only on the same tasks020k40k60kOpus 5.5Opus 5.5, Docs only, agent tasks: 32.9k characters of MCP tool text per run (95% interval 25.7k to 40.5k, 24 runs)Docs only32.9kOpus 5.5, pinright MCP, agent tasks: 10.4k characters of MCP tool text per run (95% interval 6.5k to 14.6k, 24 runs); 68% less than docs only on the same taskspinright MCP10.4k
MCP tool text per run, agent tasks, Haiku 5.5Thousands of characters the MCP tools returned per run on the 12 agent tasks, docs only against the pinright MCP server, with 95% intervals. Sonnet 5.5, Docs only, agent tasks: 24.8k characters of MCP tool text per run (95% interval 19.6k to 30.8k, 24 runs). Sonnet 5.5, pinright MCP, agent tasks: 5.3k characters of MCP tool text per run (95% interval 2.9k to 7.7k, 24 runs); 79% less than docs only on the same tasks. Opus 5.5, Docs only, agent tasks: 32.9k characters of MCP tool text per run (95% interval 25.7k to 40.5k, 24 runs). Opus 5.5, pinright MCP, agent tasks: 10.4k characters of MCP tool text per run (95% interval 6.5k to 14.6k, 24 runs); 68% less than docs only on the same tasks. Haiku 5.5, Docs only, agent tasks: 45.3k characters of MCP tool text per run (95% interval 33.3k to 60.6k, 24 runs). Haiku 5.5, pinright MCP, agent tasks: 17.2k characters of MCP tool text per run (95% interval 12.4k to 21.9k, 24 runs); 62% less than docs only on the same tasks020k40k60kHaiku 5.5Haiku 5.5, Docs only, agent tasks: 45.3k characters of MCP tool text per run (95% interval 33.3k to 60.6k, 24 runs)Docs only45.3kHaiku 5.5, pinright MCP, agent tasks: 17.2k characters of MCP tool text per run (95% interval 12.4k to 21.9k, 24 runs); 62% less than docs only on the same taskspinright MCP17.2k
MCP tool text per run, agent benchmark. Thousands of characters the docs and verdict tools returned to the agent: docs, verdicts and find-library answers.

The per-arm token intervals overlap in five of the six model and benchmark pairs; the paired intervals, which compare the same tasks, stay above zero in all six. Cache reads are most of each run's input and cost less than fresh input, so tokens and cost do not move in proportion. Data: tokens_by_model_arm.csv, from bench/tokens.ts.

With a shell, every arm ended at the same 6 alerts

The runs above give the agent Write and Read only, so it pins from memory. We ran the same 12 tasks again with Bash, each job in a disposable Linux container, with a real npm install or pip install and a build check: 216 runs over three models and three arms (no docs, hook only, MCP server and hook). Without the MCP server, the agent ran npm view or pip index versions before pinning in every run. Every arm and model ended with 6 whole-tree alerts, the ones the latest nuxt tree carries, and no direct dependency with an advisory. On these tasks, with a shell, pinright made no measurable difference to what got pinned.

Shell arm, 24 runs per arm and model, whole-tree alerts summed over the runs.
ArmSonnet 5.5Opus 5.5Haiku 5.5
No docs666
Hook only666
MCP server and hook666

The hook answered 897 tool calls and denied or asked on 48. None stopped a version with an advisory, because no agent tried to install one. 13 were false positives on npm install && npx tsc in one command, and 13 came from a release published hours earlier that deps.dev did not list yet. Hook-only runs took 15% to 21% longer than no docs. Details in RESULTS-shell.md.

The hook blocked or asked on 1.46% of 25,077 public commands

We ran 25,077 commands and manifest writes from 180 public repositories (CI steps, package.json scripts, README lines, Dockerfiles, Makefiles) through the real hook. The 1.0.0 release blocked or asked about 1.46%, and denied 0.13%. Most of those ask on purpose: a version in a variable, a git source, another package index. 32 were true positives, such as npm install -g pnpm@8 with 26 advisories in a public workflow. The 13 root causes we fixed were found on this same corpus, so there is no held-out set. We also replayed a month of our own Claude Code sessions, 154,402 distinct tool calls: after two fixes it prompted, the hook misread 10 of them, against 52 true positives and 79 asks by design.

  • Release candidate
  • After the fixes this replay prompted
Hook blocked or asked, by sourceShare of public commands and manifest writes the hook blocked or asked about, before and after the fixes this replay prompted, by source. package.json scripts: 0.27% blocked or asked before the fixes (6588 entries). package.json scripts: 0.06% blocked or asked after the fixes (6588 entries). README / CONTRIBUTING lines: 0.98% blocked or asked before the fixes (2146 entries). README / CONTRIBUTING lines: 0.61% blocked or asked after the fixes (2146 entries). Makefile / justfile targets: 5.34% blocked or asked before the fixes (711 entries). Makefile / justfile targets: 0.42% blocked or asked after the fixes (711 entries). GitHub Actions run: steps: 2.29% blocked or asked before the fixes (14040 entries). GitHub Actions run: steps: 1.54% blocked or asked after the fixes (14040 entries). Dockerfile RUN: 4.78% blocked or asked before the fixes (1171 entries). Dockerfile RUN: 4.36% blocked or asked after the fixes (1171 entries). Manifest writes: 7.60% blocked or asked before the fixes (421 entries). Manifest writes: 2.61% blocked or asked after the fixes (421 entries)0%2%4%6%8%package.json scriptspackage.json scripts: 0.27% blocked or asked before the fixes (6588 entries)Before fixes0.27%package.json scripts: 0.06% blocked or asked after the fixes (6588 entries)After fixes0.06%README / CONTRIBUTING linesREADME / CONTRIBUTING lines: 0.98% blocked or asked before the fixes (2146 entries)Before fixes0.98%README / CONTRIBUTING lines: 0.61% blocked or asked after the fixes (2146 entries)After fixes0.61%Makefile / justfile targetsMakefile / justfile targets: 5.34% blocked or asked before the fixes (711 entries)Before fixes5.34%Makefile / justfile targets: 0.42% blocked or asked after the fixes (711 entries)After fixes0.42%GitHub Actions run: stepsGitHub Actions run: steps: 2.29% blocked or asked before the fixes (14040 entries)Before fixes2.29%GitHub Actions run: steps: 1.54% blocked or asked after the fixes (14040 entries)After fixes1.54%Dockerfile RUNDockerfile RUN: 4.78% blocked or asked before the fixes (1171 entries)Before fixes4.78%Dockerfile RUN: 4.36% blocked or asked after the fixes (1171 entries)After fixes4.36%Manifest writesManifest writes: 7.60% blocked or asked before the fixes (421 entries)Before fixes7.60%Manifest writes: 2.61% blocked or asked after the fixes (421 entries)After fixes2.61%
Hook false-positive rate by source. 2.22% to 1.44% over all entries. Two fixes from the second security review ask on purpose, which puts the release at 1.46%. The corpus is CI and README commands, not agent sessions.

Latency

The hook runs before every shell command. A command that installs nothing took 121 ms at the median; one install took 358 ms with a warm cache and 2.4 s cold. With the MCP server, runs took about 2 to 4 more agent turns and 1.5 to 2 times the API-equivalent cost of a run without tools.

Hook latency, 5 runs per case, Windows, Node 24, live services. Measured once, on one machine.
CaseCachep50p95Budget
Command that installs nothing (npm test)n/a121 ms177 ms238 ms
One install (npm i express)warm358 ms372 ms1 s
One installcold2.4 s3.7 s5 s
15-line requirements.txtwarm1.0 s1.5 s1 s
15-line requirements.txtcold4.6 s5.6 s5 s
package.json with 30 new dependencieswarm1.2 s1.5 s1.5 s
package.json with 30 new dependenciescold10.9 s15.2 s45 s

The budget for a command that installs nothing is Node's own start (88 ms) plus 150 ms. A cold 15-line requirements file goes over its p95 budget.

Caveats

  • One vendor's models, all driven by Claude Code. Other agents may pin differently.
  • Tasks we wrote, and new projects only. Editing an old codebase is a different problem.
  • The headline runs give the agent no shell. With a shell, the agents looked versions up themselves and every arm ended at the same 6 alerts.
  • Advisory data from the two days of the run, 2026-11-02 and 2026-11-03. New advisories change the counts.
  • 2 runs per arm and model, so 16 to 24 runs per cell. Read the intervals, not just the bars.
  • pinright knows published advisories only. A package with no advisory on record has none reported yet.
  • The public false-positive corpus is CI and README commands. The agent-session replay covers one user's month of work.

Data and report

The technical report covers design, method, results and the security reviews. The aggregate CSVs and the scripts that draw every figure from them are in the repository. Benchmark v7 ran on the 1.0.0 code; v6, on a pre-release commit, is kept in the repository for comparison.