Are decision models deterministic?
Decision models can return probabilities without generating text, but that alone does not guarantee identical answers on repeated calls. Determinism depends on the model and serving contract; a stored response lets you replay a past decision without calling the model again.
Repeated Jev requests can change the top answer
In pgdocket's dated Jev 1.13 test, 2,000 consumer complaints went through three identical sets of requests. Only 27.5% of rows kept identical probabilities across all three runs; 95.2% kept the same top answer. The 4.8% that changed are enough to matter when reproducing an individual decision. This test does not establish the behaviour of every model or current Jev release.
The pgdocket project · Repeated-call evidence · Jev model facts
Sources: github.com
Batching and option order are separate checks
pgdocket reports about 16.5% of top answers changing when the same rows are rebatched. In a separate 1,700-question study, option shuffling barely moves overall Jev accuracy, by under two points, while the set of questions it ever gets wrong grows by 27% to 29%, beyond plain repetition. Stable aggregate accuracy can therefore coexist with changing individual answers.
Sources: github.com · arxiv.org
A vendor can promise identical requests will repeat
Finic's Grayson documentation says the same request always receives the same answers. Wording, option order, data, context and key order are part of the request. It separately says changing another question can move numerical answers slightly, under one percentage point on average. That is a vendor statement; the registry has not reproduced it.
Sources: docs.finic.ai
Store the response when a decision must be replayed
pgdocket stores versioned decisions because a fresh model call may differ from the original. Finic also tells users to retain their own request and response for audits, including its answer and request identifiers. Keep wording, option order, model identity and batching with that record, and measure repeated calls before interpreting a change as an improvement.
What is a decision model? · Decision model projects
Sources: github.com · docs.finic.ai