Decision Models / What is a decision model?
What is a decision model?
Decision models return structured answers, such as categories or scores. They can also return a yes/no probability. Jev returns these without generating text; some alternatives generate an option letter.
Definition source: Simon Willison
How decision models work
You send the context your app has, such as a message or an agent's history. A game state or web page can also be input. For a Choice question, you supply the allowed options and receive probabilities for them. You can use those probabilities to decide which answers to act on and which to send to a person or a bigger model.
Sources: simonwillison.net
Decision models and generative LLMs
A generative LLM can return structured output that your code reads. It generates its answer one token at a time. Jev returns structured answers without generating text. Together reports that Tev1 keeps a text-generation head and is trained to return one option letter.
Sources: simonwillison.net · huggingface.co
Choice, Noul and Score
Jev, and the models built to its API, take three question types. Choice returns probabilities for the options you supply. A Noul question returns a yes/no probability; the name comes from the Bernoulli distribution. A Score question returns a value along your defined scale, which can fall between the numeric levels you supply.
Sources: simonwillison.net
Decision models and System One models
TypeSafe calls Jev a "System One model", after Daniel Kahneman's split between fast, intuitive thinking and slow, deliberate reasoning. Liquid AI uses "decision models" for d1.
Sources: typesafe.ai · docs.liquid.ai
Frequently asked questions
When is a decision model good enough to replace an LLM call?
Whether a model can replace an LLM call depends on the task and error budget. In Vals AI's claim-verification test, Jev tied GPT-6-class models at 0.975. With a 1% error budget, Jev automated 95% at 1.6% error, over budget; Mercury Decide automated 94% at 0.8% error. In a study of conflicting retrieved sources, sending each decision model's least confident quarter to Gemini 3.8 Flash scored no differently from Gemini 3.8 Flash, statistically, for 8 of 11 models, at 25 to 37% of its cost. A 16-day Claude Code field report had mixed results. Of 240 scope judgments, 172 were uncertain or lacked evidence. The team removed Jev; token savings were never measured.
Evidence: Source: Vals AI · Jev evaluation 6 Oct 2026, Mercury Decide evaluation 8 Oct 2026 ↗ · Source: Khatri, Jain, Kapoor, Sah · Do We Need an LLM to Know Why Sources Conflict? · arXiv 2610.36915 · 27 Oct 2026 ↗ · Source: sash-ua · Accurate and Useless · 30 Oct 2026 ↗
Why does the answer change when I reword the question?
The linked studies tested option lists and option order. They did not systematically test rewording. Across 1,700 questions, shuffling the options moved Jev's accuracy by under two points but grew the set of questions it ever got wrong by 27 to 29%, more than plain repetition. A small hand-written field guide offers thin evidence: Jev often confidently chose the nearest option when the correct answer was missing. In a replay after the locked tests, adding a "none of these" exit raised correct handling from 11 of 14 requests to 13.
Evidence: Source: Song, Zheng, Ma, Zhou · Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias? · arXiv 2610.35826 · 26 Oct 2026 ↗ · Source: WeBuildDatabases · Jev, job by job · 29 Oct 2026 ↗
Can I trust the probabilities, including on coin-flip questions?
None of these studies tested calibration on coin flips. On other tasks, the results varied. In SILT's preregistered test of 799 unpublished yes/no questions about coding-agent work and destructive shell commands, Decider V1.1-27B was the only one of seven models to pass SILT's chosen calibration rule. Jev's result was inconclusive. So were the results for OpenAI's Decisions API and Laya. A separate banking-intent test found Jev's choice answers well calibrated but its yes/no answers underconfident on true cases. On MedQA, answers assigned at least 0.90 probability were still wrong 1.71% to 12.77% of the time, depending on the model.
Evidence: Source: SILT · Do Decision Models Know When They're Wrong? · 11 Oct 2026 ↗ · Source: ankur287 · jev-open-calibration, code and data for a Towards Data Science article · Oct 2026 ↗ · Source: Freyer et al. · Decision models vs generative LLMs on medical questions · arXiv 2610.38448 · 28 Oct 2026 ↗
How cheap is a moderation check, and where should the threshold sit?
The reports give workload-specific thresholds. Squigdle's developer reported 6 cents for about 1,000 doodle checks, about $0.00006 each, using 512×512 images plus titles and the game's own thresholds. The reported misses were occasional false alarms. This single developer's live-game report gives little evidence about other workloads. In a separate moderation simulation, a fine-tuned Laya router used a threshold set with labelled calibration data. It needed one extra moderator instead of three at 100 extra posts an hour.
Evidence: Source: James O'Halloran · Moderating 500 daily anonymous doodles with Jev · 30 Oct 2026 ↗ · Source: Enayat Hassani · Exposure-aware routing · 29 Oct 2026 ↗
Can a decision model sit in front of risky actions as a guardrail?
Two tests point both ways. In a USC input-gate test, Jev reduced answered harmful-labelled prompts from 527 to 17 out of 720, but refused 27 of 250 safe prompts, against 3 for the LLM alone. The study counted answered prompts as a proxy for harm; it did not check the completions. In Datadog's prompt-injection detector test, Jev caught 82.90% of labelled attacks; prepending a fake policy inside the untrusted text reduced that to 3.88%. That study distinguishes a classifier bypass from a full agent compromise.
Evidence: Source: Cognetti, Krishnamachari (USC) · Jev as an LLM Safety Guardrail · 2026, code and tables ↗ · Source: Datadog Security Labs · 15 Oct 2026 ↗