TypeSafe AI

Hugging Face has fixed a common interface for decision models: given a state and several typed questions, the model returns option probabilities, an ordered score or a yes/no probability without generating any text. Previously such structured judgments relied on an LLM emitting JSON, sometimes backed by constrained decoding, or on fixed-label and zero-shot classifiers. Transformers, llama.cpp and a Hub tag now follow the same contract, and compatible autoregressive language models can use the interface without retraining.

Hugging Face has fixed a common interface for decision models: given a state and several typed questions, the model returns option probabilities, an ordered score or a yes/no probability without generating any text. Previously such structured judgments relied on an LLM emitting JSON, sometimes backed by constrained decoding, or on fixed-label and zero-shot classifiers. Transformers, llama.cpp and a Hub tag now follow the same contract, and compatible autoregressive language models can use the interface without retraining.

On Movies and TV, Video Games, and Books from Amazon Reviews 2023, the study builds controlled hard candidate sets from SASRec's highly ranked negatives (candidate sizes 10 to 100) and compares Jev, a decision-oriented "System One Model" from TypeSafe AI, against SASRec, DCNv2, and pointwise and listwise Qwen2.5 7B Instruct rerankers, finding that Jev maintains strong recommendation effectiveness (NDCG@10, Hit Rate@10, MRR) while its observed serving latency grows substantially more gradually than pointwise Qwen, though it remains far above recommendation-specific models.

On Movies and TV, Video Games, and Books from Amazon Reviews 2023, the study builds controlled hard candidate sets from SASRec's highly ranked negatives (candidate sizes 10 to 100) and compares Jev, a decision-oriented "System One Model" from TypeSafe AI, against SASRec, DCNv2, and pointwise and listwise Qwen2.5 7B Instruct rerankers, finding that Jev maintains strong recommendation effectiveness (NDCG@10, Hit Rate@10, MRR) while its observed serving latency grows substantially more gradually than pointwise Qwen, though it remains far above recommendation-specific models.

The study compares a decision-only judge, JEV, which returns a verdict plus label probabilities, against sixteen generative and reward-model judges on preference, factuality, and answer-adjudication tasks with blinded human adjudication, finding it within three percentage points of the strongest comparator, GPT-6, on ordinary preference and evidence-grounded factuality at about 0.36% of that comparator's fee, with the gap concentrated in low-confidence decisions, so a frozen cascade that accepts confident verdicts and escalates uncertain ones retains about 99% of the comparator's accuracy at lower cost.

The study compares a decision-only judge, JEV, which returns a verdict plus label probabilities, against sixteen generative and reward-model judges on preference, factuality, and answer-adjudication tasks with blinded human adjudication, finding it within three percentage points of the strongest comparator, GPT-6, on ordinary preference and evidence-grounded factuality at about 0.36% of that comparator's fee, with the gap concentrated in low-confidence decisions, so a frozen cascade that accepts confident verdicts and escalates uncertain ones retains about 99% of the comparator's accuracy at lower cost.