Where this stands
HyperHat scores how engineers direct, verify and fix the work of AI coding agents across six dimensions. Each dimension is defined only as far as independent research supports it. This page sets that research out so you can judge it yourself.
HyperHat is new. We have not yet collected enough user data to publish our own validation, and we do not present research on other tools and populations as proof of our scores. What we can show is that each dimension rests on behavior that independent researchers have documented, and where that support is strong or limited. As engineers take the benchmark, we will add what we learn from real user behavior and update this page and the measurements themselves.
What the research says about working with coding agents
2.1steps per prompt
Experienced developers stay in control.
A UC San Diego and Cornell study observed 13 experienced developers working with coding agents and surveyed 99 more. They made plans before involving the agent, broke work into small steps (about 2.1 steps per prompt on average, and five to six steps before checking on larger plans), reviewed the code the agent produced, and edited it by hand when needed. They handed agents well defined work such as boilerplate, simple refactoring, tests and documentation, and kept business logic, legacy integration and architectural decisions for themselves. No survey respondent supported fully autonomous agents.
−19%measured, against +20% believed
People are poor judges of their own results.
In METR's 2025 randomized trial, experienced developers expected AI to speed them up by 24% and afterward believed it had sped them up by 20%, while measured results showed a 19% slowdown. In a Stanford study, participants using an AI assistant wrote less secure code and were more confident in it. METR has since reported that tools improved and that developers are likely faster in 2026, but the gap between feeling and measurement is the point here.
<40%comprehension when fully delegating
How someone works with AI shapes what they understand.
In an Anthropic trial of 52 engineers, those who delegated fully to AI scored below 40% on a comprehension quiz, while those who asked conceptual questions and resolved errors themselves scored 65% or higher.
48%always verify, while 96% distrust
Checking lags behind distrust.
Sonar's 2026 survey reports that 96% of developers do not fully trust AI generated code to be correct, yet only 48% always verify it. The 2025 DORA report links AI adoption to lower delivery stability when teams lack strong controls.
The six measurements at a glance
| Dimension | Behavior researchers have documented | Strength of support |
|---|---|---|
| Task Decomposition and PlanningOn your report: Task Decomposition | Experienced developers plan first and work in small steps. Planning before generating improved code accuracy in model studies. | Strong |
| Steering Precision and ContextOn your report: Prompt Quality | Specific, relevant direction improves results. Volume of generic context does not. | Strong, with a nuance |
| Verification and Code ReviewOn your report: Verification | Checking agent output separates good outcomes from bad, yet many skip it. | Strong |
| Delegation Judgment and Scope ControlOn your report: Iteration Efficiency | Experienced developers choose which work to hand to agents and keep steps small. Full delegation went with weaker understanding in one trial. | Moderate, observational |
| Hallucination Recovery and DebuggingOn your report: Recovery & Debugging | Models produce false output at measurable rates. Debugging skill showed the largest gap in the Anthropic trial. | Moderate to strong |
| System Architecture and Code QualityOn your report: Output Quality | Agents struggle with architectural decisions. Implicit quality standards drive outcomes beyond passing tests. | Moderate, indirect |
Dimension 1 of 6
Task Decomposition and Planning
On your reportTask Decomposition
What we measure
How the engineer breaks the brief into steps before and while directing the agent.
The research
In the UC San Diego and Cornell field study, developers working on larger projects built plans of 70 or more steps, then asked the agent to complete only five or six at a time before verifying. Jiang and colleagues found that having a model write a structured plan before generating code improved Pass@1 by up to 25.4% over direct generation (ACM TOSEM). That study tested models planning for themselves, so it supports the principle more than it measures human behavior.
Limits
We know of no published evidence on how much human decomposition quality changes outcomes in a timed session. Our own data can begin to answer that.
Dimension 2 of 6
Steering Precision and Context
On your reportPrompt Quality
What we measure
The quality of the direction given to the agent: specificity, constraints, relevant context and success criteria.
The research
Murr, Grainger and Gao tested five models on 104 problems under four prompt styles. Prompts that supplied test cases performed best, and removing meaningful information reduced accuracy. The developers in the UC San Diego and Cornell study gave rich, specific context such as technical terms, domain objects and file references. A 2026 preprint on 265 pull request interactions found specificity and context most strongly tied to usable code. In the Stanford study, participants who produced secure code more often supplied specifications and refined their prompts.
A nuance that shapes the definition
Gloaguen and colleagues found that repository level context files, such as AGENTS.md, did not generally improve agent task success and increased cost by more than 20% on average, though agents followed the instructions in them. Context files remained useful for documenting non standard practices. So this dimension rewards relevant, task specific direction, and does not reward volume of context.
Dimension 3 of 6
Verification and Code Review
On your reportVerification
What we measure
Whether and how the engineer checks the agent's work: running tests, reading diffs and questioning assumptions.
The research
Every developer observed in the UC San Diego and Cornell study used some form of verification, including manual testing, line by line review, running test suites and linter feedback. In the Stanford study, AI assisted participants wrote insecure solutions more often on four of five tasks while feeling more confident. A 2026 preprint identified verification as the primary predictor of whether AI generated code was adopted. A small formative study of four engineers (Catalan and colleagues) found engagement dropped during agent execution, and none of the participants correctly recalled how many functions the agent had written.
Why verification gates the score
Because trust and checking diverge so widely in surveys, a session with no logged verification says little about an engineer's judgment. HyperHat still scores the session in that case, but caps the score and flags it instead of guessing.
Dimension 4 of 6
Delegation Judgment and Scope Control
On your reportIteration Efficiency
Formerly called Iteration Efficiency, the name reports still show.
What we measure
Which work the engineer hands to the agent, how much they hand over at once, and whether they keep the agent's scope bounded.
The research
The UC San Diego and Cornell study found that experienced developers delegate selectively. They hand agents boilerplate, simple refactoring, tests, documentation and scaffolding, and keep business logic, legacy integration, architectural decisions and performance critical code for themselves. They also keep each step small enough to check. In the Anthropic trial, the engineers who fully delegated code writing, and those who gradually shifted to delegating everything, scored below 40% on comprehension, while those who stayed engaged scored far higher. UC Berkeley researchers (Cemri, Pan and colleagues) analyzed more than 150 traces across five multi agent frameworks and found that many failures began with unclear specification and weak checking, which are things a human director controls.
Limits
The evidence is observational. We know of no study that tests whether better delegation choices cause better outcomes, and the Anthropic delegation groups were small. We replaced the earlier framing of coordination efficiency because the research does not support measuring it. It supports measuring what gets delegated and how tightly it is scoped.
Dimension 5 of 6
Hallucination Recovery and Debugging
On your reportRecovery & Debugging
What we measure
What the engineer does once the agent's output turns out to be false or broken, and how well they recover.
The research
Spracklen and colleagues generated 576,000 code samples from 16 models and found commercial models invented non existent packages at least 5.2% of the time and open source models 21.7% of the time. Zhang and colleagues built a taxonomy of hallucinations in repository level code from six models (ISSTA 2025). In the Anthropic trial, the largest gap between the AI assisted and hand coding groups was on debugging questions.
Limits
Recovery overlaps with verification, since catching an error comes first. We separate them by scoring what the engineer does after a problem is detected.
Dimension 6 of 6
System Architecture and Code Quality
On your reportOutput Quality
What we measure
Whether the resulting solution meets the quality standards of the codebase, including implicit ones such as tests, documentation and formatting, and whether architectural decisions stay with the engineer.
The research
Developers in the UC San Diego and Cornell study kept architectural decisions for themselves because agents struggled with them. METR found that implicit quality requirements, such as test coverage, documentation and formatting, contributed to developers' slowdowns in 2025. DORA 2025 found that AI adoption is associated with lower delivery stability when quality controls are weak.
Limits
This dimension measures the result of the session, while the other five measure behavior. The research supports why quality matters, but not yet how it should be weighted against behavior.
What the research has not settled
- No study evaluates a six dimension composite like ours. We are combining findings from separate studies.
- Many studies are small. The field study observed 13 developers, the Stanford study had 47 participants, and the cognitive engagement study had four.
- Several studies used earlier tools. AI coding tools change quickly, and METR itself has said its 2025 results likely understate current speedups.
- Some sources are surveys, which are self reported, and some are preprints that have not completed peer review.
- None of this research tests whether these behaviors in a 30 minute benchmark predict behavior in daily work. That is what our own data must show.
How we will update this
As engineers use HyperHat, we will study how real user behavior relates to each dimension and publish what we find here, including where it does not match the research. If the data shows a dimension is not holding up, we will change it. Measurements are versioned, and each revision is logged below with the reason for the change.
Revision log
-
Version 1October 2026
Iteration Efficiency was refocused as Delegation Judgment and Scope Control, because the available research supports measuring what is delegated and how tightly it is scoped, and does not support measuring coordination efficiency.
References
- Huang, R., Reyna, A., Lerner, S., Xia, H., Hempel, B. Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025. UC San Diego and Cornell. arxiv.org/abs/2512.14012
- Gloaguen, T., Mundler, N., Muller, M., Raychev, V., Vechev, M. Evaluating AGENTS.md: Are Repository Level Context Files Helpful for Coding Agents? arxiv.org/abs/2602.11988
- Catalan, C. R., Monderin, P. N., Dizon, L. M., Kuang, E. I'm Not Reading All of That: Understanding Software Engineers' Level of Cognitive Engagement with Agentic Coding Assistants. arxiv.org/html/2603.14225
- METR. Measuring the Impact of Early 2025 AI on Experienced Open Source Developer Productivity. July 2025. metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study
- METR. We are Changing our Developer Productivity Experiment Design. February 2026. metr.org/blog/2026-02-24-uplift-update
- Perry, N., Srivastava, M., Kumar, D., Boneh, D. Do Users Write More Insecure Code with AI Assistants? ACM CCS 2023. arxiv.org/pdf/2211.03622
- Anthropic. How AI assistance impacts the formation of coding skills. anthropic.com/research/AI-assistance-coding-skills
- Murr, L., Grainger, M., Gao, D. Testing LLMs on Code Generation with Varying Levels of Prompt Specificity. 2023. arxiv.org/abs/2311.07599
- Prompt Quality and Pull Request Outcomes: A Stage Based Empirical Study of LLM Assisted Development. Preprint, 2026. arxiv.org/abs/2606.19644
- Jiang, X. et al. Self Planning Code Generation with Large Language Models. ACM TOSEM. arxiv.org/abs/2303.06689
- Cemri, M., Pan, M. Z. et al. Why Do Multi Agent LLM Systems Fail? UC Berkeley, 2025. arxiv.org/abs/2503.13657
- Spracklen, J. et al. We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. arxiv.org/abs/2406.10279
- Zhang, Z., Wang, Y., Wang, C., Chen, J., Zheng, Z. LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation. ISSTA 2025. arxiv.org/abs/2409.20550
- Google Cloud DORA. Announcing the 2025 DORA Report. cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report
- SonarSource. The AI trust gap: Why code verification matters (State of Code Developer Survey, 2026). sonarsource.com/blog/ai-coding-trust-gap
