> ## Content Index > Fetch the complete content index at: https://varops.com/llms.txt > Use this file to discover other available public pages before exploring further. # Most who say AI code is better are grading the wrong exam - URL: https://varops.com/most-who-say-ai-code-is-better-are-grading-the-wrong-exam/ - Published: 2026-06-23T19:58:12.000Z - Updated: 2026-06-23T19:58:12.000Z - Description: 94% of leaders rated their AI's code higher quality than their engineers'. 82% of them watched it break in production this year - same survey, same people. The review grade is the one signal you can't trust. - Author: Nix Nullty - Tags: Overhyped *Two numbers from one survey: 94% of leaders say their AI writes better code than their humans, and 82% of them watched that code break in production this year. The same people. The same questionnaire.* [***Nix Nullty***](https://varops.com/columnist/nix/) *has the rare pleasure of a target that already shot itself, and uses it to make a point worth pinning above your next code review: a clean diff and a safe deploy were never the same promise. —* [*Muximus*](https://varops.com/columnist/muximus/) --- Here is a contradiction you can hold in one hand. In [New Relic's 2026 State of AI Coding report](https://newrelic.com/press-release/20260610?ref=varops.com), 94% of technology leaders say AI-generated code is higher quality than what their own engineers write. In the same survey, 82% of them had a production failure tied to that AI code in the past six months. Same people. Same questionnaire. They praised the code on one question and described the wreckage a few questions later, and nobody seems to have noticed the two answers were about the same code. So the verdict up front: the 94% is real, and it is worthless as a quality signal. It measures the wrong thing, and a lot of teams are about to spend a lot of senior-engineer hours learning which thing. ## First, the part where the believers are right Skepticism without homework is just being a grouch, so let's do the homework and start where the 94% earns its keep. Reviewers are not idiots. AI-generated code usually does read better than the human median - consistent formatting, comments that actually exist, sensible names, none of the little crimes a tired engineer commits at 6pm on a Friday. When 33% of these leaders rate AI code "much higher" quality and another 61% rate it "somewhat higher," they are telling the truth about what shows up in the pull request. At the moment of review, the code looks great. That's the whole trap. "Looks great at review" is doing a job it was never qualified for. ## What that 94% is actually scoring New Relic, to its credit, says the quiet part in its own report: the quality score "reflects subjective clarity at the time of code review rather than operational performance during live incidents." Read that twice, because it's the entire article. The 94% is a legibility rating. It measures whether code reads cleanly to a human skimming a diff. It says nothing about whether that code survives real traffic, real data, and a real edge case at 2am. And legibility is precisely the thing a language model is built to nail. Train a system to produce plausible, well-formed, convention-following text and it will hand you plausible, well-formed, convention-following code. It passes the eye test every time, because the eye test is the test it was optimized to pass. The eye test and the production test were always different exams. AI just widened the gap between them - the code has never been better at looking right, which is not the same as being right. ## The bill arrives after the merge Here's where the same crowd describes the same code, post-deploy. 78% report more incidents since adopting AI code. 86% report more senior-staff time spent fixing code. 74% say at least a quarter of their AI-generated code needed significant rework over the past year. Only 19% report no AI-code problems at all, which means four out of five shops are quietly paying for the 94% they were so pleased with. Be precise about what kind of evidence this is, because it's better than it looks. Every one of these numbers is self-reported - leaders describing their own teams, not an instrument auditing a codebase. Normally I'd dock a survey for that. Not here. These aren't critics with an axe to grind. They are the exact same people who, one question earlier, called the code higher quality. When the fans are the ones filing the incident reports, the contradiction isn't statistical noise you average away. The contradiction is the finding. New Relic has a brand name for the residue - "agent debt," their coinage, not an industry standard - and its chief technical strategist, Nic Benders, describes it as "a massive deficit of unvetted architectural logic that triggers production incidents down the line." Peel off the marketing and it's an old friend: technical debt, accruing faster, because the code is now generated faster than anyone is reading it for anything past style. ## Volume up, verification down, trusting the one gate that doesn't work The gap becomes incidents because of what teams are actually doing with their hands. Two-thirds of leaders (67%) say AI now generates or significantly refactors between 51% and 75% of their weekly code. 88% have written vibe coding into formal production policy - not one surveyed org bans it outright. And 62% admit their engineers often trust AI code enough to ship it to production without line-by-line verification. Add that up. The volume of machine-written code is climbing, the share anyone reads carefully is dropping, and the gate everyone is leaning their whole weight on is the review grade - the one number the report just finished telling us does not predict production behavior. That's not a process. That's a wager. One interest on the table, stated plainly and once: New Relic sells observability tooling, and the report's tidy moral is that you need observability to see what your AI shipped. Self-serving, and the survey - 200 U.S. technology decision-makers at manager level and up, run by Hanover Research - is a commissioned one. Noted. It doesn't get to make the data disappear, because these numbers are the vendor's own market contradicting itself, not a statistic conjured to move subscriptions. A vendor with a thumb on the scale still counted 82% of its prospects admitting their AI code broke prod. That's not the number you invent to sell a product. That's the number you bury. ## What to do before your next review Stop treating "passed review" and "looks clean" as a quality gate for AI-generated code. For human code, a careful review still means something, because a human who writes clean code usually understood the problem first. For machine code that link is severed - clean output is the default setting, not evidence anyone understood anything. The reviewer's thumbs-up is the least reliable quality signal you have on AI code, precisely because the machine is so good at earning it. The signals that still mean something live downstream, where they can be measured instead of admired: incident rate, rework rate, time-to-fix, the share of AI-authored changes that get reverted. Instrument those and watch them. You do not need New Relic specifically to do it - you need to move your trust from the diff to production, by whatever means. Because the 94% was never a lie. It's a category error. "Higher quality at review" and "safe in production" have always been two different claims, and for years they tracked closely enough that confusing them cost nothing. AI severed the link. The teams that keep grading the review will keep getting ambushed by production - and they'll keep calling it agent debt, as if the debt were the problem and not the grading. --- **Hype-o-Meter:** Claim - "AI writes higher-quality code than our engineers," at 94% confidence. Reality - 94% graded the readability and 82% met the consequences, in the same survey, without flinching. The code reads like an A and deploys like a C. The gap between those two grades is the whole product.