> ## Content Index
> Fetch the complete content index at: https://varops.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# How to Talk AI Agents to Find Sneaky Production Bugs | S2E9
- URL: https://varops.com/osnt-how-to-talk-ai-agents-to-find-sneaky-production-bugs/
- Published: 2026-08-12T09:05:00.000Z
- Updated: 2026-08-12T09:05:00.000Z
- Description: A two-week unreproducible mobile bug, 30 hours of agents, and the rules Ran uses so “done” means proof — not vibes.
- Author: Ran Aroussi
- Tags: Old School / New Tech

A client mobile app at [Automaze](https://automaze.io/?ref=varops.com) had calls dropping — sometimes, only on their devices. The team circled it for two or three weeks. Almost never reproducible. Ran got pulled in as “agent whisperer,” ran two \~15-hour sessions (mostly agents working), unstuck the team, then had another agent mine the logs into an internal guide and a public write-up on *how* he talks to agents.

Not “I’m a better coder.” Better engineering standards for what counts as done.

## Don’t trust “I found it”

He trusts agents to write code. He does **not** trust “pre-existing,” “flaky,” “unrelated,” or “passing tests prove we’re good.” Treat those as hypotheses.

Example: 90 e2e tests for a feature, 85 green, five “pre-existing.” Dig anyway. The five were real bugs tied to the problem. Agents are trained to report success — mocks, skipped paths, excuses a junior would make. Force proof.

## Show me, never tell me

[Proof](https://github.com/automazeio/proof?ref=varops.com) (skill + library): every “it passes” needs video evidence — terminal, browser, mobile, desktop. Unit tests in the thousands, e2e in the hundreds, linters, profiling. Pretty code is optional once those gates are green.

Between commits: smoke with a random handful of e2e. Before merge to release/main: everything runs, everything passes.

## Confidence as a number

After “done,” ask: *what’s your confidence we can go to production?* Watch it backtrack and open items. Cap against what’s actually achievable without a human (mobile on-device might ceiling at \~90%). Don’t hand off until it’s within \~5 points of that ceiling (e.g. ≥85% if max is 90%; ≥95–98% for a fully agent-testable CLI).

## Small council of models

Match model to stakes. Judgment / review: Fable + Soul (GPT) arguing for up to five turns (skill coming public). Implementation once the plan is set: cheaper strong coders (GLM, Kimi). [Factory](https://factory.ai/?ref=varops.com) Droid makes model switching easy. Expensive for production work; worth it. Solo throwaways can stay cheap.

## Merge conflicts = stranger PR

\~30 people on a codebase — conflicts happen. Don’t “just fix the five lines.” Merge their branch into yours, then full review + full test suite as if it’s a hostile open-source contribution.

## Spec from scratch to catch drift

Before diving into code: ignore the repo for a moment, rebuild the ideal implementation plan with the agent, then diff against what’s shipped. Often the original plan was fine — **drift** from “let’s also do this” is the bug. Shortcut he uses: force “find a simpler way / reuse what exists.” Lazy engineer, less maintenance.

## Exploration vs factory

Same methodology as Cloop — but while the product shape is still fuzzy, stay in IDE land (VS Code, Cursor, Droid, etc.) with shared skills. This week’s lesson: start enforcing those standards in exploration too, not only after tickets get boring.

## The short list

1. Match model to stakes (big guns + dual review for hard / prod; cheap models for pure coding)
2. Never accept pre-existing / flaky / unrelated without investigation
3. Ask confidence % and the plan to hit the realistic ceiling
4. Control the environment (dev in the same Docker you’ll ship)
5. Observed evidence — video, logs; no summary-only “done”
6. Understand what happened (prose or code — whatever clicks)
7. PR + docs self-contained and updated on merge
8. Stay in sync with the team; resolve conflicts the hard way
9. If unverifiable, demand a precise human-only checklist
10. Methodology over a million MCPs — a handful of skills, right standards

Side note: \~300M tokens/day on his local machine alone (factory + team on top). War story from the last few days; same rules for years, sharper lately.