How to verify what your AI agent actually changed

A while back I built a companion app for a wearable device, and it passed every test I had. The simulator was green, top to bottom, run after run. Then I loaded it onto real hardware and half the screens were dead. Scrolling did nothing. The device SDK was swallowing the scroll events before my code ever saw them, and no simulator on earth was going to tell me that, because the simulator was not the device. It was a very convincing picture of the device, which is a different thing, and I had spent days trusting the picture.

I think about that build every time an AI agent hands me a summary of its own work. The answer to "how do I know what my agent actually changed" is that you do not read the summary, you read the artifact: the diff, the build output, and the thing actually running in the environment your users are in. The agent's report is the simulator. It is a rendering of the work, produced by the same process that did the work, and it is green for the same reasons the work is green. If you want to know what happened, go look at what happened.

why does my AI agent say it fixed it when nothing changed?

Because the report and the change are two different outputs, and only one of them touched your codebase. A coding agent generates a patch and then generates prose describing the patch. Both are generated. The prose is not a log, not instrumentation, not a receipt from the filesystem. What lands in your terminal is a plausible sentence written by something with every incentive to sound complete, and it can be confidently wrong while the edit never applied, or applied somewhere you did not ask for, or applied fine while the thing you are testing against never picked it up.

That third one is the cruelest, and it is the one that eats afternoons. I have watched a fix land correctly, get reported correctly, and appear to do nothing, because the process serving the app had been running since before the change and was happily holding the old code in memory. Nothing was broken. I was verifying against a stale artifact. Every round of "it still doesn't work, try again" pushed the agent to invent a more elaborate fix for a bug that had already been fixed, and the blast radius grew with each one. The report was not lying. My verification target was.

how do I check what my AI agent actually did?

Read the diff before you read the summary. Not after. The order matters more than it sounds, because once you have read a confident paragraph telling you what changed, you will see that paragraph when you look at the code. Go to the diff cold, form your own picture of what moved, and only then read what the agent claims it did. When the two disagree, you have caught something. When you read the summary first, the disagreement is invisible, because the summary has already told you what to see.

Then gate on scope. The single cheapest check in vibe coding is: did the agent touch only the files the request named? A one-line request that produces edits in six files is not necessarily wrong, but it is always a question. Somewhere in those extra files is a decision nobody asked for, and it is usually the one that bites you next week. Treat a two-hundred-line diff on a small fix as a red flag, not as thoroughness. Big diffs on small asks are where scope creep hides, and where the thing that was working quietly stops working.

And make the verification independent of the thing that produced the code. Same-author tests inherit the same blind spots as the code, which is a trap I have written about before when the tests passed and the data was fake. The same logic applies one level up: the agent's self-report inherits the agent's assumptions. If the check comes from inside the system being checked, it is not a check. Rebuild, restart, hard-refresh, hit the endpoint yourself. Watch the actual behavior in the actual environment, the way you would put the app on the actual watch instead of the simulator that was so sure it worked.

what does verifying in the real environment actually look like?

Four moves, in order, and none of them takes long once they are habit.

Read the diff first, cold, before any summary. Check the scope: only the files the request named, and if not, ask why. Rebuild or restart before you judge the result, so you are looking at the new artifact and not a cached ghost of the old one. Then exercise the actual path a user would take, in the environment they would take it in, and watch what the software does rather than what the report says it does. That last step is the whole discipline, compressed: verify in the real environment, not in the one that is convenient to check.

None of this is about distrusting AI. I orchestrate agents to build production software every day and I would not trade the leverage back. It is about knowing which signals are proposals and which are proof. The summary proposes. The diff, the build, and the running thing dispose. A vibe coder who internalizes that ordering will lose fewer afternoons than one who does not, and it costs about ninety seconds a change.

why is the vibe coding market suddenly selling agent receipts?

In one recent week I counted nearly a dozen independent tools launched with essentially the same pitch: prove the agent did what it claims. Provenance layers, diff-gating, evidence trails, tools that check an agent's claims against the actual commit. When an entire tool category appears in a fortnight, that is the market conceding something out loud, and what it is conceding here is that the report is not trustworthy on its own.

The reframe I would offer any vibe coder before they go shopping: the gate was always cheaper than the dashboard. Reading the diff is free. Rebuilding is free. Checking the file list against the request is free. Buy the tooling later if the volume justifies it, but do not buy a product to perform a discipline you have not yet adopted, because the tool will simply produce a second surface that you also have to decide whether to trust.

questions that keep coming up

The agent says the tests pass. Is that enough? No. It tells you a suite ran and matched its own assertions. Run it yourself, and look at what the assertions actually assert. Green means the code did not throw, which is the smallest possible claim.

It said it fixed it and the behavior is identical. What now? Before you re-prompt, confirm your verification target is fresh. Restart the dev server, rebuild, hard-refresh. A shocking share of "the fix didn't work" is really "I was looking at the old build." Re-prompting against a stale artifact is how a one-line fix turns into a rewrite.

How much diff review is enough on a vibe coded project? Enough to know which files moved and why. You do not have to read every line of AI-generated code, but if you cannot name the files that changed and the reason each one changed, you are not supervising the build, you are hoping.

If you are shipping a vibe coded app and you keep half-checking your agent's work by hand without ever quite trusting it, /work-with-us. Send me the build and the change you are unsure about, and I will show you where the verification gates go, so the receipts come from the artifact instead of the summary. Work with VibeKoded.

The simulator was not lying to me. It was answering a smaller question than I was asking, and answering it beautifully. Your agent's report is the same kind of green: honest about the world it can see, silent about the one your users live in. Go look at the artifact.