Changelog for the research and the tooling, newest first.
On the 77-item real test, Claude Sonnet 5.5 scored 0.987 balanced accuracy and was the only judge with true-claim accuracy 1.0. Hand-written rules scored 0.974. The confidence intervals overlap, so there's no significant difference. Sonnet's weak spot is lint and typecheck: it misses 0.381 of false claims there. Model-API cost for the run: $19.98.
An internal audit found claims the data didn't support. A no-model number-presence rule scores 77/80 on the synthetic count set, so "counting generalizes" was cut. Cells with fewer than 30 items and single-seed rows are now marked exploratory, and two seeds with empty logs are flagged.
219 items from 388 coding-agent sessions. Of 110 real "tests pass" style claims, 109 were backed by the agent's own logs. Hand-written rules score 0.922, grok-4.20 (non-reasoning) 0.932, and grok-4.7 catches 0.973 of bad claims but rejects 18% of true ones.
Two merged changes to the coding agent. Project memory now carries between sessions, with /memory to see and forget it (PR #8847). /review now applies a repository's own .supacode/review.md rules (PR #8844).