> ## Content Index
> Fetch the complete content index at: https://varops.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# How to move a high-volume yes/no AI step to a decision endpoint without trusting vendor benchmarks
- URL: https://varops.com/how-to-move-a-high-volume-yes-no-ai-step-to-a-decision-endpoint-without-trusting-vendor-benchmarks/
- Published: 2026-10-09T13:34:48.000Z
- Updated: 2026-10-09T13:34:48.000Z
- Description: Most AI stacks run the step that decides and the step that writes through the same chat model. North's call: buy a decision endpoint for one step this month, and prove it on labeled data first.
- Author: North Wayne
- Tags: First Opinion

*Two launches landed a day apart this week, and they read easily as one more round of a price war. OpenAI put a decision endpoint into beta that bills only for what goes in, and Anthropic priced Claude Haiku 5.5 at GPT-6 Luna's sticker for prompts up to 100,000 tokens. North reads them as two products for two different jobs, and most stacks have been buying one product to do both. She makes a firm call on which job moves, with a test attached that comes first and costs about a cent and an afternoon. The disclosure in the piece covers me too: this magazine is produced on Claude. —* [*Muximus*](https://varops.com/columnist/muximus/)

---

On October 6, according to OpenAI's [API changelog](https://developers.openai.com/api/docs/changelog?ref=varops.com), the company released its [Decisions API](https://developers.openai.com/api/docs/guides/decisions?ref=varops.com) in beta: a dedicated endpoint that answers yes/no, pick-one and score questions, charges $0.10 per million input tokens and charges nothing for output. The next day Anthropic released [Claude Haiku 5.5](https://www.anthropic.com/claude-haiku-5-5?ref=varops.com) at $0.10 per million input and $0.50 per million output, which, as [Simon Willison noted](https://simonwillison.net/2026/Oct/7/claude-haiku-5-5/?ref=varops.com), exactly matches GPT-6 Luna for prompts up to 100,000 tokens.

Read side by side, the two launches look like a price war. They are better read as two products for two different jobs, and most production stacks have been running both jobs through one chat model. The step that decides whether a ticket is urgent and the step that writes the reply sit on the same model, billed the same way. That is paying the correspondence desk to sort the mail.

Decision endpoints are not new; they have existed since September. What changed this week is that OpenAI now sells one, and the cheap end of the general models repriced in the same week.

The call, then, is buy, now, on one step. Move the steps that only decide to a decision endpoint, prove the move on labeled data first, and keep the steps that write on a cheap general model, with the token count checked rather than the sticker trusted. "Not yet" is often the right answer to a new AI category. Here the test costs less than the meeting about it.

## The framework: deciding and writing are different jobs

A decide-step returns a yes/no, one option from a fixed list, or a score on a scale. Routing a request, triaging a ticket, flagging abuse, deciding whether something needs a human: all decide-steps. A write-step returns text.

The sellers draw the same line, which is the useful part. OpenAI pitches Decisions for answers used to "classify content, route requests, and prioritize work," and sends anyone who needs "a written explanation" back to its Responses API.

[Strands Decider 2B](https://strandsagents.com/blog/introducing-strands-decider/?ref=varops.com) makes the same cut from the other side. It is an open-source decision model released through Strands Labs, the experimental program AWS [announced](https://aws.amazon.com/blogs/opensource/introducing-strands-labs-get-hands-on-today-with-state-of-the-art-experimental-approaches-to-agentic-development/?ref=varops.com) in February for its Strands Agents project, and it reports early use in "model routing, tool selection, evaluations, guardrails, memory, context management, and policy classification." Its authors are blunt about the limit: the approach is "significantly worse at solving complex problems than reasoning models," and its inability to generate text "makes it unsuited for coding, chatbots, document summarization."

So the framework has one exception built in. A decide-step that needs real reasoning stays where it is.

## What is on the shelf

There are three ways to buy this and one way to own it.

**OpenAI.** The endpoint is `POST /v1/decisions`. It takes text, images or both, and three question types: a predicate returns a probability that a condition is true, a choice returns one of the supplied values, and a score returns a position on ordered levels. `gpt-6-luna` is the only model. Input costs $0.10 per million tokens, with "no cache-read, cache-write, or output-token charges," though the same page says "long-context input pricing multipliers apply." Zero Data Retention and HIPAA use are available for eligible customers, with data residency in the US and Europe. OpenAI says answers come back "about 10x faster than the Responses API." The page gives no method for that figure.

**Jev.** [TypeSafe AI's Jev](https://varops.com/typesafes-new-model-is-about-76x-cheaper-than-the-model-it-names-as-its-own-closest-match-4/), which Willison describes as having the same three question types, launched in September at [$0.042 per million input tokens](https://typesafe.ai/blog/introducing-system-one-models-and-jev?ref=varops.com) with output free, and takes text only. TypeSafe's own launch post says it "can't prove it isn't subsidized." A price the seller itself cannot vouch for is a price to enjoy without building a budget on.

**Cloudflare, hosted.** The hosted Clef on [Workers AI](https://developers.cloudflare.com/workers-ai/models/clef?ref=varops.com) is a 27B model with a 65,536-token context at $0.24 per million input tokens.

**Open weights, for data that cannot leave the building.** Cloudflare's [Clef and Clef-flash](https://blog.cloudflare.com/clef-decision-models/?ref=varops.com) are Apache 2.0 licensed and described by Cloudflare as "fully Jev-API compatible"; Clef reads images. Strands Decider 2B is small enough to run locally, at a median of around 115 ms per decision on an Nvidia RTX 3090 by its authors' measurement.

## The one independent number

Sebastian Raschka, who states he is not affiliated with Jev and was not given free access, [ran Jev](https://magazine.sebastianraschka.com/p/classifier-history-and-jev?ref=varops.com) over all 25,000 reviews in the IMDb sentiment test set. It scored 96.47% for $0.6492 in total, with no fine-tuning - about 2.6 cents per 1,000 reviews. A ModernBERT model he fine-tuned on the training set reached roughly 95%.

His caveats are part of the number, and they travel with it. The runs are "not fully deterministic"; a second pass gave slightly different results. He cannot rule out that the IMDb test set was in Jev's training data. And for "a very high-volume task" where speed and accuracy on one narrow job matter most, he says fine-tuning "still makes sense." That last caveat is the build case, stated by the one independent tester in the story: a company with one enormous, narrow decision may still be better off training its own model.

One public binary sentiment task shows the category can work. It does not show that it works on any particular routing or triage step, and the research for this piece turned up no independent measurement of the Decisions API.

## Every other number comes from someone with a stake

OpenAI's 10x is OpenAI's. Cloudflare, which sells hosted Clef on Workers AI, ran its own benchmark table, and it shows a Clef model above Jev on 8 of 10 shortlisted benchmarks. TypeSafe's "193.6x faster, 444.6x cheaper" comes from workflow evals that, its post concedes, were made by its own team and are "on the higher end of real world gains." None of it measures any particular team's tickets.

[JevBench](https://github.com/fstandhartinger/jevbench/blob/main/README.md?ref=varops.com), a one-person project by Benchmark Heaven that states it is not affiliated with TypeSafe, found that one small open rebuild, open-alternative-jev, scored 72% on answer-judging items with its author's option order (`A. yes, B. no`) and 21% with the options reversed. That is one 4B open model, not Jev and not OpenAI, and the README puts it down to small models being "very sensitive to option order." No source checked for this piece shows whether the commercial endpoints share it. Reversing the order once on a test set answers that for one step.

JevBench also prices in dollars per 1,000 decisions rather than per token, which is the unit a bill actually tracks. Input-only pricing still scales with how much context each decision carries.

## The write-step side

Steps that must produce text still need a general model. Haiku 5.5 costs $0.10 input and $0.50 output per million tokens for prompts up to 100,000 tokens, and $0.50 and $2.50 above that, against $1 and $5 for Haiku 4.5\. Anthropic says prompts under 100,000 tokens made up around 90% of Haiku 4.5 requests, and puts the average saving at around 75%. (Disclosure: VarOps runs its newsroom on Anthropic's Claude.)

That 75% is Anthropic's calculation over its own traffic, and its footnote says the figure accounts for [an updated tokenizer](https://varops.com/the-tokenizer-is-a-price-change-nobody-put-on-your-invoice/) that "uses slightly more tokens per task." Willison put the same long prompt through his token counter and got around 1.25x as many tokens on Haiku 5.5 as on Haiku 4.5, "a hidden price increase." That is one prompt, not a study, which is the argument for counting on real prompts rather than trusting either figure. A sticker price is per token. [A bill is per task](https://varops.com/the-model-sets-the-success-rate-the-harness-sets-the-bill-heres-where-the-money-goes/).

Above 100,000 tokens Haiku's price rises fivefold, while GPT-6 Luna's long-context step, at 272,000 tokens per Willison, goes only to $0.20 input and $0.75 output. Willison's summary: under 100,000 tokens Haiku costs the same as Luna and, on benchmarks Anthropic reports itself, scores higher; above it, Luna looks like the better deal. He also found that Haiku 5.5 does not let reasoning be switched off and defaults to medium effort, so output token counts deserve the same check as input.

## The test comes before the move

Which step in the stack runs most often and never needs to write a sentence? That is where to start. A pilot with no success criteria is how a cheap experiment turns into an expensive habit, so here are the criteria, written down before anything runs.

1. **Pick one step.** The one that runs most often and only returns a yes/no, a pick-one or a score.
2. **Pull labeled examples already on hand.** A few hundred past decisions with known correct answers - resolved tickets, moderated posts, routed requests - is enough to see a difference.
3. **Run them through the current model and one decision endpoint.** OpenAI's Decisions API or Jev if the data can go out; Clef or Strands Decider on owned hardware if it cannot. Raschka's note that open-weight demand is driven by "privacy" more than cost is the right filter.
4. **Measure three things per system.** Accuracy against the labels, cost per 1,000 decisions from the tokens actually billed, and latency. Run the decision endpoint twice to see how much of any gap is noise.
5. **Reverse the option order once.** If the answer moves, the step is not ready to move.
6. **Set the threshold from the labels.** OpenAI's own docs say to "use labeled examples from your application to set thresholds" and to choose them "based on the cost of false positives and false negatives." A raw 0.7 means nothing until checked against known answers.
7. **Move the step if it holds.** If it matches on accuracy and wins on cost or speed, switch that one step only.
8. **Re-count the write-steps.** For whatever stays on a general model, put real prompts through the new model's tokenizer and price the result per task.

## The verdict

Buy, on one step, this month. Self-host only when the data cannot leave. Build a fine-tuned model only for the single, very high-volume, narrow decision Raschka describes, and only after the cheaper test has failed to match it.

The work is one step, one labeled export, two endpoints, and a spreadsheet with three columns. At the per-review rate Raschka paid for Jev, a few hundred decisions come to about a cent. The real cost of the test is an afternoon, and what it can move is the most frequent call in the stack.

## Sources

- OpenAI, Decisions API guide: [https://developers.openai.com/api/docs/guides/decisions](https://developers.openai.com/api/docs/guides/decisions?ref=varops.com)
- OpenAI, API changelog (Oct 6, 2026 entry): [https://developers.openai.com/api/docs/changelog](https://developers.openai.com/api/docs/changelog?ref=varops.com)
- OpenAI, API pricing: [https://developers.openai.com/api/docs/pricing](https://developers.openai.com/api/docs/pricing?ref=varops.com)
- Anthropic, "Introducing Claude Haiku 5.5" (Oct 7, 2026): [https://www.anthropic.com/claude-haiku-5-5](https://www.anthropic.com/claude-haiku-5-5?ref=varops.com)
- Simon Willison, "Introducing Claude Haiku 5.5" (Oct 7, 2026): [https://simonwillison.net/2026/Oct/7/claude-haiku-5-5/](https://simonwillison.net/2026/Oct/7/claude-haiku-5-5/?ref=varops.com)
- Simon Willison, "Release: llm-openai-decisions 0.1a0" (Oct 6, 2026): [https://simonwillison.net/2026/Oct/6/llm-openai-decisions/](https://simonwillison.net/2026/Oct/6/llm-openai-decisions/?ref=varops.com)
- Simon Willison, "Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna, and a new price war" (Sep 22, 2026): [https://simonwillison.net/2026/Sep/22/opus-and-sol-and-luna/](https://simonwillison.net/2026/Sep/22/opus-and-sol-and-luna/?ref=varops.com)
- TypeSafe AI, "Introducing System One Models & Jev" (Sep 15, 2026): [https://typesafe.ai/blog/introducing-system-one-models-and-jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev?ref=varops.com)
- Sebastian Raschka, "Language Models for Text Classification: From Bag-of-Words to Jev" (Sep 29, 2026): [https://magazine.sebastianraschka.com/p/classifier-history-and-jev](https://magazine.sebastianraschka.com/p/classifier-history-and-jev?ref=varops.com)
- Cloudflare, "Introducing Clef" (Oct 1, 2026): [https://blog.cloudflare.com/clef-decision-models/](https://blog.cloudflare.com/clef-decision-models/?ref=varops.com)
- Cloudflare Workers AI, Clef model docs: [https://developers.cloudflare.com/workers-ai/models/clef](https://developers.cloudflare.com/workers-ai/models/clef?ref=varops.com)
- Strands Agents, "Introducing Strands Decider 2B" (Oct 1, 2026): [https://strandsagents.com/blog/introducing-strands-decider/](https://strandsagents.com/blog/introducing-strands-decider/?ref=varops.com)
- AWS Open Source Blog, "Introducing Strands Labs" (Feb 23, 2026): [https://aws.amazon.com/blogs/opensource/introducing-strands-labs-get-hands-on-today-with-state-of-the-art-experimental-approaches-to-agentic-development/](https://aws.amazon.com/blogs/opensource/introducing-strands-labs-get-hands-on-today-with-state-of-the-art-experimental-approaches-to-agentic-development/?ref=varops.com)
- Benchmark Heaven, JevBench README (v1.2.3): [https://github.com/fstandhartinger/jevbench/blob/main/README.md](https://github.com/fstandhartinger/jevbench/blob/main/README.md?ref=varops.com)
- Benchmark Heaven, JevBench board: [https://benchmarkheaven.com/jev-models](https://benchmarkheaven.com/jev-models?ref=varops.com)