When an AI agent tells me it is done, I treat that as a claim. Then I run a check. This post explains what I mean by a check, what makes a good one, and two places where a check did a job no agent could do reliably.
A check is a calculator. An agent is a sample.
Ask a calculator for 2 + 2 and it says 4. Ask it a million times and it says 4 a million times. That is a check: a program that gives the same answer for the same input, every time.
Ask an agent the same question and you get its judgment. Its judgment is usually right. But each answer is a sample. Some share of the time it is wrong, or it skims past a typo, or it says “all good” because that is the likely thing to say. And you can’t tell which answers those are without checking them.
Asking a second agent to review the first doesn’t fix this. It just draws another sample. So wherever I can write down what “correct” means, I write it as code.
What makes a good check
Not every script is a good check. The ones that earn their keep share six traits:
- Deterministic. Same input, same answer. No model inside.
- Fast and cheap. Cheap enough to run every single time, not once a week.
- Pass or fail, with a reason. Not a score. “Line 12: 40 not in approved facts” beats “looks mostly fine”.
- Reads the real output. The actual files, the actual page, the actual resume. Not a test copy or a summary of it.
- Fails loudly. A failure stops the work and says so. Silence never counts as a pass.
- Written once, run forever. I pay for it one time. Every later run is free.
Here are two cases from my own systems.
Example 1: a move that broke none of 10,656 links
When I rebuilt my agents system, I moved all my notes into a new folder layout: 1,100 live files plus a larger archive. My notes link to each other a lot. Move a file and every link to it breaks.
I could have asked agents to fix the links. That would mean thousands of small hand edits. Each one is a chance to miss a link, or to “fix” it to a path that does not exist. Then I would need another agent to check the work, which is just another sample. I needed every link right, not most of them.
So the move was a script, with no model in it. It works in four steps:
- Plan. List every move as old path to new path.
- Rewrite. Change every link that points at a moved file, by rule, and log each edit.
- Check. Re-scan every link and confirm it resolves. Every file the tool did not edit must match its fingerprint (a hash of its contents) exactly. Every file it did edit must equal the old file plus its logged edits, nothing more.
- Cut over, meaning swap the new folder in for the old one, only if step 3 says PASS. Any mismatch prints FAIL, and my working folder is never touched.
Before the real move, the tool rehearsed the whole thing into a scratch copy and checked it. It checked 10,656 links between my files. It made 1,950 link rewrites in 410 files, and proved each one still pointed at the same file. Without those rewrites, 1,832 links would have broken. After the rehearsal, 0 new links were broken, and every unchanged file matched its fingerprint. The real move had to pass the same check before the swap.
One honest note: 1,960 links were already broken before the move. The check reported them and left them alone, so the move made nothing worse.
Later, agents built a smaller move tool for single files. Their work showed why checks matter. Two cases:
- A guard that “passed” but was not safe. A worker reported that the tool never deletes a top-level notes folder when it cleans up empty ones. A separate checker re-ran it with my real config instead of the test config. With the real config, the guard failed. It was minor and nothing was lost, but the claim was false, and the tests had said it was true.
- Tests that guarded nothing. A checker deleted one branch of the code in a copy and ran the tests. All 18 tests for that part still passed. No test covered that branch. Breaking code on purpose is the fastest way I know to test the tests.
In both cases an agent said done. The catch came from running the real code, not from anyone’s opinion of it.
Example 2: tailoring a resume without inventing a fact
I use a model to tailor my resume to a job posting. It reorders bullets, trims, and rewrites the summary. That is useful work for a model. But a model writing about me will sometimes improve on the truth: a number that sounds good, a skill I don’t have, a more senior title.
So I don’t ask the model to be careful. A script checks every fact in a tailored resume against a list of facts I approved: my master resume, plus a few achievement lines I vetted from source records. The script checks:
- Numbers. Every number in the draft must appear in the approved facts.
- Job headings. Each title, employer and date range must match word for word, in the same order.
- Bullets. Each bullet must match an approved line, or be a shortened version of one: words can be cut, never added.
- Skills. Each skill must already appear in my approved facts.
Anything not found is blocked, and the reasons go back to the model for another try. After two failed tries, it ships a base resume I already checked, instead of the draft. A draft that fails the check never ships.
Here is what a catch looks like. This is an illustration, built from a line on my public work history:
How often does this happen? In one test run, I tried seven setups (different prompts and starting resumes), 30 drafts each: 210 drafts. The script blocked 20 of them, for 24 bullets that matched no approved line. The setup I use had 0 blocked. The more open setups had up to 8 out of 30. As my own design notes put it, prompt rules alone are the weakest guard.
One part is not a script. The summary is new prose, so there is no approved line to match it against. A second model reviews it, and in that same run it flagged 12 more drafts. One earlier catch was a made-up “staff engineer” summary. Its objections also go back to the model. That reviewer is a sample too. So I keep its job small: everything that can be checked by code is checked by code first.
The rule I follow now
When the answer must be right every time, I write a check, and the agent’s “done” is only the start. Agents are great at doing the work. Checks are how I know it was done right.
If you want to see how I measure the agents themselves, I wrote about that here.