Hyperhat Hyperhat

Menu

Research For Teams Docs Blog

Your agent says it's done. What evidence should you ask for?

What should you ask an AI agent for before you merge its work? A practical evidence format: the requirement, the change, the checks, and the gaps.

Before merging agent written code, ask for a short evidence packet. State the requirement, show the change, list the checks that were actually run, and flag what wasn't checked, rather than accepting a confident summary as proof.

Engineering teams are shipping more code through AI agents every quarter, and the agent's own summary has quietly become the default signal for whether that work is ready to merge. A summary describes intent. It says nothing about whether the requirement was actually met, whether the tests still test the right thing, or whether an important case was simply never checked.

Why a Confident Summary Isn't Enough

"Implemented the feature and added tests" is a useful heads up. It isn't enough to make an acceptance decision.

The reviewer still needs to know which behavior changed, what the tests actually establish, and what remains unchecked. A summary is a map to the evidence. It is not the evidence itself.

That distinction matters most when the agent can edit both the implementation and the tests used to judge it. A passing check deserves a second question: does this still test the requirement we started with?

Connect Every Requirement to a Check

Take a hypothetical checkout change: prevent a double click from creating two orders. A disabled button is visible evidence of one behavior. On its own, it doesn't establish that two requests arriving close together can't create duplicate orders.

The reviewer needs to see what happens at the order creation boundary. Which check exercises repeated requests? Does it run against the real implementation, or replace that implementation with a mock? A check of button state and a check of order creation answer different questions. Write the requirement beside the check that supports it. When no check supports an important requirement, the gap should be obvious.

A diff that touches files the request never implied is worth a question before it's worth a read. Unexplained scope is where most surprises hide.

What the Test Diff Reveals That the Code Diff Doesn't

Read the test diff alongside the implementation diff. Look for changed expectations, removed assertions, new skips, and mocks introduced around the behavior being changed. None of that is automatically wrong, but the reason should be visible. A weaker assertion shouldn't quietly become the new definition of success.

A pattern worth watching for: a new test gets added and passes, while the original failing test quietly picks up a skip. The result looks green and the summary sounds confident, and the bug is still there. Reading the test files before the implementation diff catches this faster than reading the code first.

The check also matters more when it comes from somewhere the agent didn't shape. A test written and graded by the same run that did the work is close to grading its own homework. The strongest checks exist before or apart from the task: a test you wrote yourself, a diff read against the original request, or a review pass that had no hand in the implementation.

The Five-Part Handoff That Makes Evidence Inspectable

A completion format any team can adapt, kept short, linking to the underlying output rather than filling the pull request with logs:

  • Requested behavior: what should now be true, using the original requirement as the reference.
  • Change: the relevant diff and any deliberate change in scope.
  • Checks: the commands actually run, their results, and which requirements they support.
  • Test changes: what changed in the checks and why. Flag removed or weakened coverage.
  • Gaps: what wasn't run, what couldn't be verified, and what still needs a person's decision.

This is a proposed format, not a claim that every change needs a formal report. A small copy change might only need a diff and a preview. A payment, permission, or data migration change needs evidence suited to its consequences.

Why Gaps Need to Stay Visible

"Could not run the integration test because the service was unavailable" is useful information. "All done," with that limitation left out, is a different handoff entirely. Don't collapse implemented, checked, and accepted into one status.

A step report that says what it expected against what it actually found, rather than just "done," turns a claim into something you can check. A step that returns four of forty processed, stopped at page one, surfaces a problem immediately instead of three steps later.

A second AI reviewer can help find issues. It can't settle whether the original request was right, or whether the checks actually cover it. Give the reviewer the requirement and the evidence, and keep a person accountable for the acceptance decision.

Agree on the Evidence Before the Work Starts

A completion checklist is less useful if everyone sees it for the first time at the end. Before delegating, name the behaviors that matter, the boundaries the agent should preserve, and the evidence you expect back.

One cheap habit pairs well with this format: run the one input or click path that motivated the task yourself, not the full suite, just that case. It takes a minute, and it catches a fix that solved a different problem than the one reported.

Then hold the same acceptance bar whether the reviewer wrote the request or is seeing it for the first time. If understanding the result means replaying an entire conversation, the handoff needs work.

The Habit Behind the Handoff

This is part of why I'm building HyperHat. Co-Director is our focus for helping engineers catch drifting agents while they're still working, not after the fact. The benchmark supports that work by making direction and verification habits visible, with a session replay you can actually walk through.

An evidence packet is a habit a team can adopt today, without waiting on us or any other tool. The point isn't to automate the acceptance decision. It's to make that decision easier to inspect, while keeping the engineer who makes it accountable for it.

Frequently Asked Questions

Why isn't a passing test enough to prove an agent's change is correct?
Because the agent can edit both the implementation and the tests used to judge it. A passing check deserves a second question: does it still test the requirement that was originally asked for, or has the test quietly changed along with the code?

What should a completion handoff from an AI agent actually include?
Five things: the requested behavior stated against the original requirement, the change itself, the checks that were run and what they support, what changed in the tests and why, and the gaps, what wasn't run or couldn't be verified.

Does the review process change when the reviewer didn't write the original request?
Yes. A reviewer seeing the request for the first time can't tell whether it was satisfied or merely addressed unless the request itself carries acceptance criteria a stranger could check. The same acceptance bar should still apply either way.

What's a cheap way to sanity check an agent's claim that something is fixed?
Run the one input or click path that motivated the task yourself, not the full test suite, just that case. It takes about a minute and catches a fix that solved a different problem than the one reported.

Can a second AI reviewer replace this evidence packet?
No. A second AI reviewer can help surface issues, but it can't settle whether the original request was the right one or whether the checks actually cover it. A person needs to stay accountable for the acceptance decision.

Blog

View all posts