My first question about Jev was straightforward: why introduce another model if a cheap language model can already return the decision my application needs?
That question deserves a practical test. It also deserves a workload that gives Jev's distinctive interface something useful to do. Choosing one of three labels is a limited way to assess a model designed to return many typed decisions over shared context.
I used Codex to extend our initial pilot with two tasks an engineer could recognize in a real project: routing a support request across 77 intents, and answering 17 questions about a complete contract. We compared Jev with GPT-6 Luna and Qwen 3.7 Flash, then checked what happened when we bundled questions instead of making separate calls.
The goal is practical guidance for engineers applying these models, not a scientific ranking. The useful outputs are the tradeoffs, failures and request designs worth testing in your own application. The method and results and reproduction bundle include the inputs, expected labels, harness and recorded responses.
Give both interfaces a useful workload
Jev accepts state and typed questions, returning decisions with probabilities. TypeSafe describes parallel output as a central advantage and demonstrates choices among many candidates. These are two different dimensions: the number of options in one question, and the number of questions over one input. Vercel exposes the latter through its evaluation endpoint. TypeSafe's launch, Vercel evaluation API
Our original six-model test used 18 synthetic examples with only three or four candidate labels. Jev matched 35 of 36 judgments and Luna 36, but two task families were too easy to distinguish the models. That pilot did not test either dimension well. Its original records remain available.
For the September 25 follow-up, I chose Luna because it had the highest agreement in that pilot and Qwen because it was cheapest. Each received the same text, questions and label definitions as Jev. The language models used strict JSON output with reasoning disabled. Crucially, they could also return all 17 contract answers in one call. There was no requirement to generate explanations or make one call per answer.
We used public test-set annotations, frozen before inference. There was one call per example and configuration, without a repetition loop: 77 support requests and 12 contracts per model. Two contracts also received the separate-question treatment. That made 369 initial API calls across three models, not hundreds of repetitions of the same examples.
Routing across 77 intents
BANKING77 contains online banking queries labeled by intent. We selected one test query per intent using a fixed hash rule and presented all 77 labels every time. The labels' names supplied their descriptions; we did not tune them against the answers.
| Model | Responses received | Label matches among responses | Median response time | Reported market cost |
|---|---|---|---|---|
| Jev | 73/77 | 59/73 | 363 ms | $0.005243 |
| GPT-6 Luna | 77/77 | 63/77 | 1,424 ms | $0.001886 |
| Qwen 3.7 Flash | 77/77 | 60/77 | 1,004 ms | $0.000986 |
These are first-pass results. Jev's four missing responses were HTTP 503 service errors, not wrong labels. Its cost column includes the reported costs of successful calls; the failed calls supplied no cost metadata. Latency covers successful complete responses. All returned answers had the required shape and an allowed label. Run details
The cost result is useful precisely because it differs from our tiny first pilot. Luna reported cached input on 76 of 77 routing calls; Qwen on 74. The fixed taxonomy made much of the prompt reusable. These are observed costs with that caching behavior, not a comparison of uncached list prices. In a project with a stable routing prompt, I would measure that effect before assuming which model is cheaper.
The disagreements also expose work that belongs to the application team. A request about a transfer that has not appeared can plausibly sit near several intents. One selected query, “what is my card PIN”, has the dataset label get_physical_card; all three models chose change_pin. I would review that annotation before treating the disagreement as evidence against every model. We kept the original labels in the scores rather than correcting them after seeing the outputs.
My takeaway for a routing project is to inspect the taxonomy and ambiguous examples alongside model choice. One query per intent gives coverage, not a dependable accuracy estimate for a production inbox. Nor does this test cover selecting several labels at once: each request still chooses one intent from many.
Seventeen judgments over one contract
ContractNLI provides contracts with 17 fixed hypotheses, labeled as supported, contradicted or not mentioned. We selected 12 test contracts spread across document lengths, from about 3,200 to 25,200 characters. Each request supplied the complete document and asked all 17 questions. We tested classification only, not evidence-span extraction or legal advice.
Jev completed eight of the twelve bundled calls on the first attempt. Luna and Qwen completed all twelve. For a comparison of decisions on identical inputs, the table below uses only the eight contracts all three completed, containing 136 judgments:
| Model | Matches on the same 136 judgments | Median response time | Reported market cost for eight contracts |
|---|---|---|---|
| Jev | 104/136 | 497 ms | $0.001483 |
| GPT-6 Luna | 111/136 | 2,181 ms | $0.002722 |
| Qwen 3.7 Flash | 101/136 | 2,898 ms | $0.001268 |
Across all twelve documents, Luna matched 172/204 labels and Qwen 158/204. Jev initially delivered 104 matching decisions, 32 disagreements and 68 missing decisions because four bundles failed. Restricting the table to completed documents does not remove that coverage problem. No model matched all 17 labels on any of its completed first-pass documents. Complete results, including failures and class breakdowns
That is a meaningful engineering tradeoff. Jev returned successful bundles in roughly half a second. Luna matched more annotations on the shared documents. Neither observation makes an unattended contract-review workflow ready to ship. For that project, I would surface the supporting clauses and exceptions for review before letting the decisions drive consequential actions.
Bundling helps the cheap models too
On two preselected contracts, we also asked the same 17 questions through separate calls, sequentially per model. One ran separate calls first; the other ran the bundle first. For the contract where every first-pass call succeeded, the comparison was:
| Model | 17 separate calls: total response time | One bundled call | Cost reduction from bundling | Label matches: separate → bundled |
|---|---|---|---|---|
| Jev | 6.63 s | 0.36 s | 7.3× | 10 → 11 |
| GPT-6 Luna | 24.15 s | 2.38 s | 12.5× | 14 → 14 |
| Qwen 3.7 Flash | 17.56 s | 3.15 s | 5.5× | 11 → 14 |
These are request-design observations on one contract, not general speedup factors. Separate-call time is the sum of response times; parallelizing those calls could lower wall-clock latency. Cache behavior is included in cost. Some individual answers changed even when the total number of matches stayed the same.
On the second contract, both language models again used less reported inference cost and less summed response time when bundled. Jev's bundle and one of its separate calls failed, so there is no complete first-pass comparison for it there. Paired records
The practical lesson is to test a compact batch of decisions before introducing a loop of API calls. Jev makes that design explicit, but cheap structured-output models can benefit from it too. Check whether batching changes the answers as well as the bill.
Availability is part of the decision
Jev had nine HTTP 503 responses across its 123 initial calls: four routing requests, four document bundles and one separate question. The other two models had none in this run. This describes the endpoint experience during our test window; it is not a long-term service-reliability estimate or proof of an architectural cause.
After the run, we made one additional attempt for each failed request, with the original request unchanged. Seven succeeded and two contract bundles still failed. Those recovery records are separate: they do not overwrite the first-pass results or establish the latency of an immediate retry policy. After recovery, Jev had answered every banking query, matching 61/77 labels, and ten contract bundles, matching 128/170 delivered judgments.
For a real integration, I would test bounded retries, a fallback or review queue, and a clear incomplete-result state. A failed 17-question bundle leaves more missing decisions than a failed single question. The surrounding code needs to handle that explicitly.
What LangChain adds—and what to try next
LangChain's September 20 Jev-as-a-judge experiment examined repeated judgments on five frozen weather-agent runs. It reported much lower continuous-score variance for Jev, alongside strong agreement with a human reviewer on binary decisions. That makes evaluator consistency worth investigating, but repetitions of five examples do not establish broad accuracy, and low variance does not demonstrate calibrated probabilities.
Our tests answer a different question: what happens when an engineer gives these APIs a routing taxonomy or several decisions over a real document? Public datasets let us make that comparison inspectable, but they may have appeared in training data, and we did not independently adjudicate the entire sample. We also have not measured sustained throughput, confidence thresholds or how performance changes across a sweep of label counts.
For a project, I would start with a small collection of representative inputs and expected outcomes the team has reviewed. Run the same task through Jev and a cheap structured-output model. Batch related questions, record cache behavior, inspect disagreements, and test what happens when a request fails. Measure the complete workflow against its latency budget and the cost of a bad decision.
Jev earned further investigation here through the speed of its successful responses. Cheap models remained credible alternatives, and their cost depended on how we used them. The choice should follow the workflow you can actually operate.
Sources
- Our practical tests: method, results and run records, September 25, 2026; reproduction bundle.
- Original six-model synthetic pilot, September 24, 2026.
- Casanueva et al.: Efficient Intent Detection with Dual Sentence Encoders, 2020; BANKING77 dataset, CC BY 4.0.
- Koreeda and Manning: ContractNLI, 2021; dataset, CC BY 4.0.
- TypeSafe: Introducing System One Models & Jev, September 15, 2026; models and confidence.
- Vercel: evaluation API.
- LangChain: Jev-as-a-Judge for Agent Evals, September 20, 2026; experiment repository.