Overhyped

Two launches this week sold coding models on DeepSWE, where every run starts from an untouched repo. Hand the same agents work that only looked finished and they lost 20.6 to 53.5 points – and the top two from a clean start fell to third and fourth.

OpenAI shipped 722 AI-written math papers with Lean proofs, then withdrew three a day later. A green check proves the formal file. Here is what it leaves out, and four questions to ask before it moves a decision.

Fireworks' 40% token cut came in at 23% in the first independent test we found. The saving is real. Whether it lowers your bill depends on who you were buying Kimi K3 from.

Ninety-seven percent in constructed runs. One in 417 real pull-request pairs. The paper reports both numbers, says neither is a frequency, and names the execution records nobody has that would settle it.

1Password's 26% clean-fix rate averaged in trials that forbade the agent from testing and trials told to patch it wrong. Trail of Bits reran the same published data and got 86% - then published its own human failure rate next to it.

A June preprint reports a 4-5x speedup. A case study seventy-five days later reports agents take longer than experienced researchers. Neither team buried anything and neither is wrong - they are reading different clocks. Only one of the three ever reaches a headcount line.

A vendor-commissioned evaluation scored 720 prompt injections at zero. Weeks later a public exploit chain beat the same mode. Both numbers are correct.

GitHub meters Copilot in tokens and still bills by the seat. Uber's AI budget went in four months. The unit in our planning documents stopped measuring the work.

Microsoft documented the cloud dependency, the prompt egress and the identifier collection on the page a user would actually read. The compute really is local. What no page mentions: the value stamped into the pixels is a per-generation identifier issued by the server that read your prompt.

Pull requests per Linear workspace are up 111% in two years. Over the last year of that window, time on existing work held or drifted up - and Linear says so itself. Anthropic puts its own AI speed-up at less than double. The number that got funded is not the number being measured.

The "2.6x cheaper language" number now living in Google's AI answers measures token density, not what a coding agent actually costs. Somebody finally ran the real evals.

OpenAI ran its own model on ARC-AGI-3 twice and got 13.3% and 38.3% - same tasks, two API settings. A benchmark scores the harness as much as the model, which is why a leaderboard gap is a bad way to choose a vendor.

A benchmark score is not a capability. A new audit read 2,385 agent-benchmark runs and found most scored by gaming the test - and the one question to make a vendor answer before their leaderboard number moves your money.

The viral "OpenAI cut Codex context" was a bug fix on three of eight models - and the 272k everyone panicked over was never the ceiling.

Claude Code sends 33k tokens before your prompt. It's real - and it's the cheapest line on your bill, because it gets cached at 0.1x. The numbers that actually cost you were four screens down in the same study.

Everyone is using AI for everything? Microsoft sells Copilot and still can only find about a third of us. The adoption number in your board deck is a bubble reading - here's the one the data supports.

94% of leaders rated their AI's code higher quality than their engineers'. 82% of them watched it break in production this year - same survey, same people. The review grade is the one signal you can't trust.

Rio de Janeiro's "sovereign AI" called itself Nex 79% of the time with the system prompt stripped — because it is, mathematically. The weights are 60% Nex-N2-Pro, and the math remembers where you got the parts.

METR ran the only controlled trial on AI developer productivity and found developers 19% slower. Then the follow-up broke — here's why the "2x faster" claim has no verified source and what that means for your next board meeting.