> ## Content Index > Fetch the complete content index at: https://varops.com/llms.txt > Use this file to discover other available public pages before exploring further. # Honeycomb shipped twice as fast with AI and tripled its incidents. Neither number says anything about risk. - URL: https://varops.com/honeycomb-doubled-its-shipping-rate-and-published-the-incident-count-the-number-nobody-has-is-the-rate/ - Published: 2026-07-20T09:32:17.000Z - Updated: 2026-07-20T09:32:17.000Z - Description: Honeycomb went from 30 to 74 merges a day with AI - and published the incident count too, from 18.5 a quarter to 53. The number it still can't produce is the one you need. - Author: Rex Factor - Tags: Proof of Work *Everyone publishes the multiplier. Almost nobody publishes what it cost. Honeycomb did both this month - 30 to 74 merges a day, and an incident count that went from 18.5 a quarter to 53 - which makes it the most useful AI productivity document of the year and, on closer reading, an admission.* [**Rex**](https://varops.com/columnist/rex/) *went through both posts with a calculator. The finding isn't that Honeycomb's numbers are bad. It's that the best-instrumented team in the business, at a company that sells instrumentation, still cannot tell you whether each change it ships is riskier than it used to be. Neither can your vendor.* *— Muximus* --- Honeycomb's engineering team went from roughly 30 peak-weekday merges in early 2025 to roughly 74 in April 2026\. In the same window, incidents went from a 2024 baseline of about 18.5 per quarter to 32 in Q1 2026, then 53 in Q2. Read that second sentence again, because it is the one that almost never gets written down. I have read a lot of AI productivity case studies. The throughput number is always there. The cost line is usually missing, and when it appears it is usually a reassurance rather than a number. Honeycomb published both, on 16 July, in a two-part report-out with the methodology attached and a running commentary on why you should not take the headline at face value. It is a more careful accounting than most companies publish, or could. It also cannot answer the question you actually have. Neither can anyone else. That is the finding. ## By the numbers - Peak-weekday merges: **\~30** (early 2025) to **\~74** (April 2026). Peak day hit 70 again on 18 June. - Incidents per quarter: **\~18.5** (2024 baseline), **32** (Q1 2026), **53** (Q2 2026). - New lines of code carrying AI attribution, June 2026: **at least 82.6%**. Highest month on record. - Codebase: **\~972,000** lines (end 2024) to **2,096,286** (6 July 2026). It doubled in sixteen and a half months. The previous doubling took nearly three years. - Autonomous agent merges, June: **70** of the month's merges (**8.4%**) - accounting for about **5,500 lines**, or **0.3%** of the codebase. - PR auto-approval at Honeycomb today: **zero**. One disclosure before we go further, because it belongs where the numbers land and not in a footnote. Honeycomb sells observability. The thesis running through both posts - that fast feedback loops and production observability are what make high AI throughput survivable - is also a description of its own product category. Every figure above is first-party telemetry, not independently audited. The numbers are considerably more self-critical than most vendor-adjacent data ever gets. They are still the vendor's numbers, and you should weigh them knowing it. ## The part they got right, stated fairly Honeycomb has not had a spectacular AI-caused failure. No deleted database. No corrupted user data. They are careful to say this is neither luck nor evidence that AI is safe. It is the return on defensive design that predates the agents: bulkheads between components, review, deploy trains, feature flags, least-privilege access. Their instruction to anyone without that is one sentence and I would put it on a wall. If you do not have it in place, "that's the thing to fix before you scale up AI usage, not after." Their stated goal is not what a vendor would promise, either. It is keeping each failure "cheap to contain, not holding the count flat." Nobody's roadmap slide says that. It is the correct target, and the rest of this piece assumes it rather than argues with it. ## Four numbers, four questions There is no single AI percentage in these posts, and that is deliberate. Honeycomb publishes four denominators for April 2026, because they answer four different questions: | What you're asking | April 2026 floor | | ------------------------------------------------------ | ---------------- | | How many engineers use AI at all? | \~95% | | What share of merged PRs had AI in the commit history? | at least 63% | | What share of new lines came from AI-touched PRs? | at least 75% | | What share of merged PRs included an AI review step? | 100% | Any one of those could be quoted as "our AI number." Each would be defensible. Each would be misleading. Every one is a floor, not an estimate, because git only records what the tooling chose to record. Honeycomb's example is the best argument against your own dashboard I have seen this year: one of their heaviest Claude Code users by token volume has **zero** AI-attributed commits. Twenty-two sessions, 199 million tokens, thirty days, invisible to git, because a co-author trailer was switched off at tool level. If that engineer is invisible, others are. And the warning that travels furthest: set to loose confidence, Honeycomb's vendor dashboard "cheerfully reported a majority of PRs as AI" before any calibration had been done. The published floors come from hand-calibrated git history instead - attribution recovered from squash-merged branches, from deleted branches via the GitHub API across all 7,952 PRs, and from session telemetry for engineers whose git record showed nothing at all. That is what it costs to know your own AI number. Three reconstruction passes over every pull request you have ever merged. Whatever your [vendor dashboard](https://varops.com/every-instrument-on-your-dashboard-was-supplied-by-the-vendor/) is showing you, it did not do that. ## The rate nobody has Here is Honeycomb's reading of its own incident line: it tracks change volume. More changes shipped means more chances for a defect to land in the batch, at whatever your baseline defect rate happens to be. They cite the VOID report and Google's DORA research for change being the leading driver of incidents industry-wide, and they conclude that "the law of large numbers caught up with us." On the comparison they make, that holds up. Q1 2026 ran 1.7 times the 2024 incident baseline against throughput at roughly 2.5 times its own early-2025 baseline - which, they note, "looked sub-linear" for exactly one quarter before Q2 withdrew the comfort. Q2 came in at 2.9 times baseline incidents against throughput still around 2.5 times. Two quarters in, the two lines move together at about the rate you would expect. Now look at what that comparison is made of. Two multiples, measured from two different baselines, in two different units, over two different periods. It is enough to say the lines move together. It is not enough to produce a rate. And a rate is the entire question. Not "did incidents go up" - of course they went up, you shipped more. The question every operator actually has is narrower: **is any given change now more likely to break something than it was before the agents arrived?** In the language your CFO uses: is the defect rate per unit of work holding, or are you buying throughput with quality and booking it as volume? You cannot get there from what was published, and the obstruction is denominators. The throughput multiplier is peak-weekday merges. The incident multiplier is a quarterly total. Those do not divide. It compounds inside the throughput series itself, too: the June data cut excludes mechanical bots from every throughput denominator while the April figures earlier in the same post do not - a discontinuity Honeycomb flags itself. There is no published series of incidents per change, per week, at constant scope and constant scope of measurement. So nobody can compute the rate. Including Honeycomb. To their considerable credit, they do not pretend otherwise. They report the growth. They decline to pin it on AI, noting that "broader AI-causation narratives tend to be self-flattering, whichever direction they point." They say it is too early to tell whether AI assistance in debugging reduces incident severity once something breaks. They say they are leaning on better automated preflight checks to try to bend the curve down. Part two leaves the incident line filed under open questions rather than resolved ones. That is the state of the art. The best-instrumented public account of an AI throughput gain, from a company that sells instrumentation, reports its quality line as a raw count in units that will not divide into its throughput line - and says so. Anyone showing you a cleaner answer than that is showing you something they did not measure. Two more admissions from that section deserve to survive the summary, because no chart carries them. The volume "showed up as real strain on the teams absorbing it, not just as a line going up on a chart." And on the metric itself: "Throughput is an input metric, not the product... Throughput going up while user outcomes plateau or decline is, by definition, enshittification." A company reporting a 2.5x that calls its own headline metric an input, not the product, is a company worth reading carefully. ## How much of it is even new work Honeycomb cannot tell you how much of the gain is additional output versus the same output relabeled. They say so directly. Peak-weekday merges carrying no AI attribution held steady at 25 to 30 across the whole window. On their best days, engineers did not slow down to make room for the agents. But at the weekly org-wide level, human-attributed commit volume fell from about 120 a week before February to about 60-80 a week from February to April, while AI-attributed volume grew from near-zero to 150-200. Some of that is new capacity. Some of it is the same engineer moving from typing code to driving an agent, so work that used to carry a person's name now carries a bot's. Honeycomb's position: it cannot separate the two "without a controlled experiment we don't have." Their autonomous workflow makes the point sharper than any argument would. The autobot - which commits with no human in the coding loop - went from 3 merges in April to 70 across four weeks of June, 8.4% of merges. Those 70 merges account for roughly 5,500 lines. About 0.3% of the codebase. Eight percent of the merges. A third of one percent of the code. A merge-count gain and a code-volume gain are not the same measurement, and if you are about to set a PR-count target for your own team, that is the sentence to sit with. ## The base rate, and why I am not going to oversell the gap Honeycomb sets its own result against the industry base rate and concedes the position. Credit where it is due - and this is also the comparison most likely to get quoted with its caveats stripped off, so let me do the stripping-back myself. DX, which sells developer-productivity measurement, published preliminary data from a longitudinal study drawing on a random sample out of 400 companies between November 2024 and February 2026\. AI usage rose 65%. Pull-request throughput rose about 7.8%. DX's headline: "AI productivity gains are 10%, not 10x." Honeycomb's own response, in that light: "We're the outlier case here, not the median one." In comparable units, Honeycomb's peak-weekday throughput rose about 150% against DX's roughly 8%. That is a wide gap, and it is narrower than it first looks, for two reasons that cut against the argument I am making. The windows barely overlap where it counts. DX's data runs to February 2026\. Honeycomb's step change begins in February 2026\. The median in that sample is largely a pre-inflection base rate, so this is not a clean same-period comparison of one company against four hundred - and DX labels the data preliminary and has updated it since. The measurements are also not the same measurement. Honeycomb's figure is peak-weekday merges in one monorepo, hand-calibrated. DX's is PR throughput across a sample of companies. Putting them side by side is useful for orientation and would be an error to treat as a controlled comparison - which is precisely the error Honeycomb spends its first post warning against. What survives is still the useful question. Some organizations get a step change out of these tools and most get single digits, and the difference is [not which model they bought](https://varops.com/a-new-model-wont-save-you/). Everyone in both populations had frontier models available. ## What the difference is actually made of Honeycomb's answer, and the thesis of the second post: AI amplifies whatever practices you already have. "It can make a dysfunctional org more dysfunctional, or it can bring out the best in an org that has high autonomy, ownership, and feedback loops." The prerequisites they name are specific and almost entirely unglamorous. Continuous delivery. CI that is fast and legible to an agent. Feature flags. Code-ownership practices. Blameless incident analysis. Codified rules an agent can verify against. Mediated access so an agent reviews with the context a human would have had. Production observability that links shipped code back to the PR that created it. It would be tidier to tell you all of that existed before the AI arrived. It did not, quite - and this is where I part company with the clean version of the story. Honeycomb dates a major CI investment to late March 2026, mid-ramp, and notes that some of the tooling was itself built with the new capacity. So the honest shape is a compounding loop, not a static precondition. Enough substrate to start safely, then some of the new capacity spent on more substrate. That is harder to copy than a checklist and a more accurate description of what happened. It also means the entry price is not "have everything first" - it is "have enough to not hurt yourself, then spend the winnings on the guardrails rather than on more throughput." Most teams will spend the winnings on more throughput. The blunt version, from part two: "Nothing Honeycomb, or Intercom, or anyone else on a stage tells you about AI will land if your starting substrate is unhealthy." ## The tell The clearest evidence that Honeycomb means the substrate argument is a thing it has not switched on. Intercom - which Honeycomb reports hit the same February 2026 inflection, and which it describes itself as following by a few months - is at 19.2% AI auto-approval of pull requests as of April. Honeycomb is at zero. Deliberately. Their reasoning is that the compliance question is necessary but not sufficient, because the technical preconditions sit underneath the procedural ones. Without the codified rules, the agent-legible CI, the observability loop, and some way for humans to stay aware of what is shipping when they are not the ones gating it, the same 19% means something else entirely: "it's measuring rubber-stamping against weak constraints, not safe automation against strong ones." Two companies can report identical auto-approval percentages and be running two completely different risks. That is worth remembering the next time a peer quotes theirs at you. ## Three questions Someone is going to show you a throughput multiplier this quarter. A vendor, a peer company, a board member with an article. Three questions make it useful or make it evaporate. **What is the denominator?** Peak or average? Which population? Measured how? Is the figure a floor or an estimate? Honeycomb answers all four about its own throughput, and the answers cut the headline roughly in half. **Where is the quality line, and is it in units that divide into the throughput line?** Not whether they have one - whether the two can be put over each other to give a rate. Honeycomb published a quality line, which is more than almost anyone does, and still cannot produce that rate. If a vendor hands you one, ask what they measured to get it. **What was already in place, and what got built along the way?** If the answer is continuous delivery, fast CI, feature flags and observability that ties production behavior to a specific change, the multiplier is downstream of years of unglamorous work. Some of which, in Honeycomb's case, was still being built while the multiplier arrived. If your own organization cannot answer those three questions about itself today, that is the project. The model rollout is not. --- **Sources** - Honeycomb, ["30 to 70 PRs a Day: How We Managed to Not Wreck Our Systems"](https://www.honeycomb.io/blog/30-70-prs-day-how-we-managed-not-wreck-systems?ref=varops.com), 16 July 2026 - Honeycomb, ["AI Amplifies Your Existing Practices: Lessons from Our Shift to an AI-First Strategy"](https://www.honeycomb.io/blog/ai-amplifies-existing-practices-lessons-ai-first-strategy?ref=varops.com), 16 July 2026 - Justin Reock, DX, ["AI productivity gains are 10%, not 10x"](https://newsletter.getdx.com/p/ai-productivity-gains-are-10-not?ref=varops.com), 11 March 2026, updated 28 April 2026