ClaimCheck
A small benchmark for one question: when a coding agent says "tests pass" or "lint is clean," does its own session log back that up? Data, rubric and harness are public at github.com/clickmind/claimcheck.
The dataset (v1.2)
- 219 items: 110 real claims, 109 built from real items by rule (wrong run, inflated count, run edited to fail).
- Taken from 388 session logs of AI coding agents working on one TypeScript product, October 2026.
- 178 test claims, 31 lint, 10 typecheck. One annotator (me), with model help.
Result 1: the agents' claims held up
Of 110 real success claims, 109 were backed by the agent's own logs. The one exception called eslint "clean" when the log showed 4 warnings.
Result 2: checkers on v1.2 (219 items)
| Checker | Accuracy | Catch rate on bad claims | True-claim accuracy |
|---|---|---|---|
| Always trust the agent | 0.498 | 0 | 0.991 |
| Keyword (exit 0, no "failed") | 0.712 | 0.464 | 0.964 |
| Hand-written rules | 0.922 | 0.891 | 0.955 |
| grok-4.20 (non-reasoning) | 0.932 | 0.909 | 0.955 |
| grok-4.7 | 0.895 | 0.973 | 0.818 |
Source: README, v1.2 table. The rules baseline is close to how labels were made, so its score overstates how hard the task is for rules on new data.
Result 3: frontier judges on the 77-item real test
| Judge | Accuracy [95% CI] | Note |
|---|---|---|
| Claude Sonnet 5.5 | 0.987 [0.958, 1.0] | Balanced accuracy. True-claim accuracy 1.0; misses 0.381 of false lint/typecheck claims |
| Hand-written rules | 0.974 [0.930, 1.000] | CIs overlap with Sonnet: no significant difference |
| grok-4.20 (non-reasoning) | 0.974 [0.937, 1.000] | |
| grok-4.7 | 0.909 [0.847, 0.971] | |
| Qwen2.5-0.5B + LoRA (ours) | 0.831 [0.753, 0.905] | Below rules and frontier judges |
Sources: 1a0bb5d (Claude arm, $7.84), 6d8f74b (Grok arms, $12.14). Model-API total for judges run #1: $19.98.
What did not work
- The small fine-tuned model (Qwen2.5-0.5B + LoRA) stays below hand-written rules on the real test.
- Generic synthetic count-mismatch training did not transfer to real claims (real count catch 0.28, 4 seeds). A generator that copies how real agent claims look reached 0.875 in the next round.
- Counting on clean logs turned out to be script-solvable: a number-presence baseline gets 77/80. I removed the claim that counting "generalizes."
- 1.5B models were not feasible on the 8 GB CPU sandboxes.
Source: README after audit commit f9ecbac ("demote overclaims per audit").
How the work is run
Every experiment is pre-registered in a commit before any training or API call, then results are reported against it, including the hypotheses that failed. Spend is logged per run.
Limits
The bad examples are synthetic, there is one annotator, it is one codebase, and it is 219 items. Labels were revised in v1.1 after seeing model output; the README lists each change.
Next: ClaimCheck World pre-registered, not run
Real tiny vitest/pytest repos run in sandboxes with injected chaos (flaky tests, skipped files, timeouts, truncated logs), labeled by the actual test reporter output. Pre-registration 36997d8.