Give Your Skill a Report Card

Ask an AI to rate its own output on a 1-to-5 scale and you'll get a number — but not a trustworthy one. AI can't reliably tell the difference between a four out of five and a five out of five; when pushed for a score, it makes one up. If you want a grading loop you can actually trust, don't ask for a score. Ask for pass or fail.

Why numeric scores don't work as evals

A 1-to-5 rating feels rigorous because it produces a number, but the number is mostly noise. The model has no stable internal sense of "this is a 4.2" versus "this is a 3.8" — those distinctions are finer than what it can consistently judge, so consecutive runs on the same output can drift up or down for no real reason. That makes numeric self-scoring useless as a stop condition: you can't tell whether "4" means "good enough to ship" or "needs another pass."

Binary checks don't have that problem. "Does this include a call to action?" and "Are there filler words?" are questions with an actual, checkable answer. The model isn't estimating a fuzzy scalar — it's applying a concrete rule to concrete text.

Build an evals.md of pass/fail checks

For any skill that produces written or structured output, write down the checks that define "done" as explicit, binary items:

No filler words?            ✓ / ✗
Has a clear call to action? ✓ / ✗
Matches the target voice?   ✓ / ✗

Each check should be answerable by looking at the output — no judgment calls about degree, only about presence or absence. If a check feels like it needs a "somewhat" answer, it's not specific enough yet; split it into something that resolves to yes or no.

Grade it with a separate, clean-context agent

The check that matters as much as the checks themselves: don't have the same agent that wrote the output also grade it. An agent grading its own recent work carries the context of everything it just did — its own reasoning, its own assumptions — and that bias leaks into the grading, the same way a person struggles to proofread their own writing right after finishing it.

Spin up a separate agent to run the evals, with a clean context window that has no memory of how the output was produced. It sees only the evals file and the output, and answers each check independently. That separation is what makes the grading trustworthy instead of self-congratulatory.

The loop

1. Editing agent produces (or revises) the output.
2. Grading agent, in a fresh context, runs every check in evals.md and reports pass/fail per item.
3. If any check fails, send the failures back to the editing agent and have it revise specifically to address them — not a general rewrite, a targeted fix.
4. Repeat until every check passes.

This isn't a one-shot review — it's a loop, and it can take more than one pass. One real run of this pattern took five loops before every check passed, running unattended: nobody sat and watched each round happen. The value of the loop is exactly that it doesn't need a human in the middle of every iteration — the grader either passes it or kicks it back.

Setting up your own evals file

• Write checks specific to what "good" means for that particular skill's output — a newsletter-editing skill and a code-review skill need entirely different checklists.
• Keep every check binary. If you're tempted to write "tone is roughly right," rewrite it as something checkable, like "avoids first-person plural" or "no sentence over 30 words."
• Store the evals file next to the skill so it travels with it and stays in sync as the skill's behavior changes.

Power tricks

• Always keep editing and grading agents separate. Never let the same context that produced the output also judge it.
• Turn fuzzy checks into binary ones. "Quality: good/bad" is a hidden score — break it into the concrete traits that actually make it good or bad.
• Let the loop run unattended. The whole benefit is not babysitting every revision — check back once every item passes.

Resources

• Claude docs — Agent Skills overview
• Anthropic — Claude Code best practices