Decision Models / JevBench: scores, SemIf and Jev
JevBench: scores, SemIf and Jev
JevBench is Benchmark Heaven's independent evaluation of Jev-compatible decision models. Florian Standhartinger runs the project. It measures decision quality and calibration alongside speed and cost, with separate boards for self-hosted systems and hosted APIs.
Sources: benchmarkheaven.com · benchmarkheaven.com · benchmarkheaven.com
Board published 1 Nov 2026 (v1.8.6)
The board published 01 Nov 2026 reports 337 measured models and 217 ranked systems on its open-weights board. Jev 1.13.0 is an unranked reference at 77.1 Capability. The first four rows have open-weight status Unknown on this board; check each model's licence before choosing how to run it.
| Rank | Model | Capability | Board's open-weight status |
|---|---|---|---|
| 1 | Baseten Mammoth-1 (27B) | 85.2 | Unknown |
| 2 | KnowLine 31B-Gen1 | 85.2 | Unknown |
| 3 | Fastino GLiDE 1.2 | 84.9 | Unknown |
| 4 | H2O-Lightning-31B v1.0.7 | 84.3 | Unknown |
| 5 | Perplexity Decider v1.1 27B | 82.7 | Yes |
| 143 | SemIf (formerly OpenJev), Qwen3.5-4B | 55.4 | Yes |
Sources: benchmarkheaven.com
What JevBench measures
Each request supplies application state and a bounded question. Intelligence measures chance-corrected decision accuracy, with tier weighting and a public/sealed gap adjustment. Calibration scores the probability distribution. Capability is their arithmetic mean, with ranks restricted by cost and median-latency caps. The hosted API board leads with Composite, a weighted harmonic mean of Intelligence, Calibration, Speed and Cost with penalties. It also reports dollars per 1,000 decisions and p50 latency.
Sources: benchmarkheaven.com
Where SemIf, OpenJev and TypeSafe's Jev sit
SemIf is the project formerly called OpenJev, independent of TypeSafe. Its Qwen3.5-4B row ranks #143 on the 01 Nov 2026 v1.8.6 board, with 55.4 Capability, 27.1 Intelligence and 83.6 Calibration. Jev 1.13.0 scores 77.1 Capability as the unranked reference. TypeSafe's Jev 1.14.0 appears as a second unranked reference here and is ranked on the API board. These scores describe the recorded versions and setups.
Sources: benchmarkheaven.com
Timing and cost rules
Method v1.7 counts a decision taking more than 10 seconds as wrong and gives it zero calibration credit. Exactly 10 seconds passes. Self-hosted timings are measured over loopback; ranking uses an estimated response latency, including network and serving overhead, of twice measured p50 plus 150 ms. That estimate has not yet been validated against public endpoints. From v1.8, self-hosted cost uses estimated GPU time, capped at the previous list-price estimate. Hosted rows retain published API prices.
Sources: benchmarkheaven.com
Artificial Analysis Decision Index
Artificial Analysis runs a separate evaluation. Its v1.0 index, launched 13 Oct 2026, combines four evaluations covering knowledge and hallucination, reasoning, tool calling and safety classification. Cost, speed and calibration are reported alongside it. Subquadratic's 22 Oct release post quotes 65 for SubQ Snap 1.1 on that index. Its scale and tasks differ from JevBench's, so scores cannot be compared directly.
Sources: artificialanalysis.ai · subq.ai
Jev Decision Index
The community Jev Decision Index, hosted by multimodalart on Hugging Face, has its own tasks and scoring rules. For the v0.4 text board, Full score combines 60% private tests of the same skills and 40% private tasks from new domains, each mapped onto the public scale. Public text results are displayed separately and excluded from Full score. Scores and ranks cannot be carried between this index, Artificial Analysis and JevBench.
Sources: huggingface.co
Jev alternatives · Published model prices · What is a decision model?