What it does. Every prompt tweak in a long-running agent looks like an improvement in the moment. The only way to know is a graded run against fixed inputs. Loopkit already ships
.claude/agents/verifier.md
— that is your grader. Do not rebuild it.
Each stage writes to disk. No stage holds the whole run in context.
Highlights
- 20-100 cases is enough for a signal. More is nice, not required.
- Include known-hard cases, edge cases, and a couple of trivial ones as sanity anchors.
- Freeze the file. Rev the eval with a suffix (
inputs-v2.jsonl
) when you change it. Never edit in place — you lose the baseline.
- Same temperature every run (usually 0 for evals).
- Same seed / model version.
When to use it. Build a repeatable evaluation harness for agent work: fixtures, scoring, and a regression loop, from the 54-skill loopkit engineering pack.
Install
npx skills add Archive228/loopkit --skill eval-harness
Or copy the skill folder manually:
git clone https://github.com/Archive228/loopkit.git
cp -r loopkit/skills/eval-harness/ ~/.claude/skills/eval-harness
Source: Archive228/loopkit — 754 stars, MIT, last pushed 2026-07-14.
