# Jev in two practical decision workflows

Protocol frozen September 25, 2026 before the scored calls. The purpose is to help engineers try these models in real projects, not to establish a scientific ranking.

## Workloads and selection

- **Support routing:** 77 distinct BANKING77 test requests, one per intent. Choose the query with the smallest SHA-256 of a fixed seed plus its text within each intent. Every request presents all 77 candidates. Descriptions are dataset intent names with underscores replaced by spaces; there is no hand-tuned intent taxonomy, few-shot prompting or retrieval shortlist.
- **Document review:** 12 complete ContractNLI test contracts, chosen at the midpoints of twelve strata sorted by document character length. Ask all 17 published hypotheses in one request: Entailment, Contradiction or NotMentioned. This measures the classification part of ContractNLI; it does not evaluate evidence-span extraction or perform legal review. No document is truncated. The selected contracts range from 3,244 to 25,167 characters.
- **Request design:** On two preselected contracts (indices 3 and 8 of that length-ordered sample), also ask the same 17 questions in separate calls. One document runs separate calls before its bundle, the other afterwards. Compare the total inference cost, summed request latency and correctness of processing the same document. The separate baseline is sequential per model, not an optimized parallel fan-out.

The fixtures, expected labels, selection seed, source identifiers and file hashes are frozen in `cases.json`. The expected answers come from the dataset authors, not a model or this article's author. We do not independently re-adjudicate the entire sample. These are public test sets, not private held-out production data; training-data exposure cannot be ruled out. One example per banking intent and twelve contracts provide engineering diagnostics, not population accuracy estimates. Decisions within a contract are related.

## Models and requests

The follow-up uses Jev, GPT-6 Luna and Qwen 3.7 Flash. Luna had the highest correctness and Qwen the lowest cost in our original six-model pilot. Selecting them makes this a focused follow-up rather than a rerun of every model.

Both interfaces receive the same state, question wording and label definitions. Jev uses native choice questions. The chat models use a strict JSON schema mapping each question key to one label, with reasoning disabled. All models can return the 17 answers together; chat models are not forced into separate calls or explanations. Jev returns its native probabilities as well, but these are not required from chat models because this workflow consumes decisions. We do not compare probability quality or calibration.

One call per case/configuration, no repetition loop. There are 123 requests per model: 77 routing calls, 12 bundled contract calls, and 34 separate-question calls. That is 369 total requests, covering 89 distinct inputs and 281 unique labeled decisions per model; the 34 split decisions are a paired request-design comparison, not additional examples.

The harness launches one call per model together for each job, rotates launch order, and waits for all three before the next job. There is never more than one in-flight request for a given model. This is an application-path comparison through Gateway, not an isolated architecture benchmark or a throughput load test. Provider defaults apply except reasoning `none`, strict output schema, and a 2,048-token completion limit for chat models. Each request has a 45-second timeout and no application retry. Gateway may route or fail over internally. No warm-up results are discarded.

## Scoring and accounting

Count exact label matches against the frozen dataset annotations. Missing, invalid or failed answers count as incorrect in the attempted-decision denominator. Report valid response counts separately. For document bundles, also report contracts for which all 17 answers match. Report median and p95 complete-response latency for successful calls, not time to first token. Show class-wise outcomes for ContractNLI so the common label cannot conceal weaker performance on contradictions.

Use Gateway-reported `marketCost` for comparisons and retain actual account `cost` separately. A zero account charge does not establish permanent free inference. Costs include the provider's observed cache behavior; the separate/bundled comparison does not normalize cache hits. Report costs for each workload, rather than combining them into a misleading universal price. Stop before the next three-call round at $0.50 observed market cost, on an authentication/payment restriction, or when successful calls omit cost metadata. This is not a provider-enforced hard spending cap.

Every run directory saves a manifest, runner snapshot, selected model catalog, exact requests (without credentials), response content, Jev distributions, usage, routing, cost and timings. No fixtures, prompts or labels are changed after observing results.

## Reproduction

Download the original BANKING77 `test.csv` and `categories.json` from its `banking_data` directory, and the ContractNLI archive linked below. Save them as `banking77-test.csv`, `banking77-categories.json` and `contract-nli.zip` in a private scratch directory. Run:

```sh
python3 scripts/research/prepare-jev-practical.py /path/to/downloads
node --env-file=/private/path/gateway.env scripts/research/compare-jev-practical.mjs
node scripts/research/summarize-jev-practical.mjs content/research/jev-practical-2026-09-25/runs/RUN_DIRECTORY
```

The environment file needs `AI_GATEWAY_API_KEY` or a current `VERCEL_OIDC_TOKEN`. Do not place it in the reproduction bundle. Dataset sources may change; compare hashes against the frozen fixtures. The saved subset itself is sufficient to rerun inference.

## Post-run availability follow-up

Nine Jev requests returned HTTP 503 in the first pass. We added one separate recovery pass using their exact recorded requests, after the first pass had finished. Seven succeeded and two failed again. No original record or score was replaced. This addition is an operational diagnostic, not part of the original one-call protocol. Run `node --env-file=/private/path/gateway.env scripts/research/recover-jev-practical.mjs RUN_DIRECTORY` to reproduce that one recovery pass, then summarize again. The recovery script refuses to overwrite an existing pass.

## Attribution and sources

- Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson and Ivan Vulić (2020), [Efficient Intent Detection with Dual Sentence Encoders](https://arxiv.org/abs/2003.04807). [BANKING77 data](https://github.com/PolyAI-LDN/task-specific-datasets), CC BY 4.0. We selected a subset and converted it to the common evaluation format; original text and labels are retained. See `BANKING77-LICENSE.txt`.
- Yuta Koreeda and Christopher D. Manning (2021), [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://aclanthology.org/2021.findings-emnlp.164). [Dataset and download](https://stanfordnlp.github.io/contract-nli/), CC BY 4.0. We selected a subset and converted its classification task to typed questions; document text, hypotheses and labels are retained. See `CONTRACTNLI-LICENSE.txt`.
- [Vercel evaluation API](https://vercel.com/docs/ai-gateway/modalities/evaluation).
- [TypeSafe model documentation](https://docs.typesafe.ai/models).
- [LangChain's separate Jev judge experiment](https://www.langchain.com/blog/jev-agent-evals-langsmith), September 20, 2026. Context for the article, not the source of our measured results.


# Observed results — September 25, 2026

First pass: 369 requests. Recovery: 9 additional attempts for service failures, 7 successful. Known market cost across both passes: $0.031261247. Failed calls without price metadata are not assumed free.

## First-pass workloads

### BANKING77, all 77 candidates

| Model | Completed calls | Matches / attempted decisions | Matches / delivered decisions | Median / p95 ms | Reported market cost | Missing cost records |
|---|---:|---:|---:|---:|---:|---:|
| typesafe-ai/jev | 73/77 | 59/77 | 59/73 | 363 / 659 | $0.005243238 | 4 |
| openai/gpt-6-luna | 77/77 | 63/77 | 63/77 | 1424 / 3042 | $0.001886335 | 0 |
| alibaba/qwen3.7-flash | 77/77 | 60/77 | 60/77 | 1004 / 1638 | $0.000986190 | 0 |

### ContractNLI, 17 questions per bundle

| Model | Completed calls | Matches / attempted decisions | Matches / delivered decisions | Median / p95 ms | Reported market cost | Missing cost records |
|---|---:|---:|---:|---:|---:|---:|
| typesafe-ai/jev | 8/12 | 104/204 | 104/136 | 496.5 / 576 | $0.001482516 | 4 |
| openai/gpt-6-luna | 12/12 | 172/204 | 172/204 | 2168 / 2378 | $0.004410320 | 0 |
| alibaba/qwen3.7-flash | 12/12 | 158/204 | 158/204 | 3011 / 3519 | $0.002014570 | 0 |

### ContractNLI, same eight first-pass completed contracts

| Model | Completed calls | Matches / attempted decisions | Matches / delivered decisions | Median / p95 ms | Reported market cost | Missing cost records |
|---|---:|---:|---:|---:|---:|---:|
| typesafe-ai/jev | 8/8 | 104/136 | 104/136 | 496.5 / 576 | $0.001482516 | 0 |
| openai/gpt-6-luna | 8/8 | 111/136 | 111/136 | 2180.5 / 2378 | $0.002722165 | 0 |
| alibaba/qwen3.7-flash | 8/8 | 101/136 | 101/136 | 2898 / 3469 | $0.001268310 | 0 |

Attempted-decision denominators include missing decisions; delivered-decision denominators exclude service failures. Successful-response latency excludes failed requests. All successful calls produced valid output shapes. No first-pass completed contract had all 17 labels correct.

## Contract classes, all twelve requested documents

Counts are matches / expected decisions, including missing decisions from failed calls.

- typesafe-ai/jev: Contradiction 8/15, Entailment 58/100, NotMentioned 38/89
- openai/gpt-6-luna: Contradiction 12/15, Entailment 87/100, NotMentioned 73/89
- alibaba/qwen3.7-flash: Contradiction 9/15, Entailment 92/100, NotMentioned 57/89

## Paired request design

| Contract | Model | Bundled matches /17 | Separate matches /17 | Bundle / separate successful calls | Bundle / separate summed latency ms | Bundle / separate reported cost |
|---|---|---:|---:|---|---|---|
| contractnli-test-83 | typesafe-ai/jev | 11 | 10 | 1/1; 17/17 | 355 / 6627 | $0.000161700 / $0.001175748 |
| contractnli-test-52 | typesafe-ai/jev | 0 | 9 | 0/1; 16/17 | 1224 / 7755 | $0.000000000 / $0.002050566 |
| contractnli-test-83 | openai/gpt-6-luna | 14 | 14 | 1/1; 17/17 | 2378 / 24146 | $0.000246195 / $0.003083350 |
| contractnli-test-52 | openai/gpt-6-luna | 16 | 13 | 1/1; 17/17 | 2139 / 23225 | $0.000408195 / $0.005837350 |
| contractnli-test-83 | alibaba/qwen3.7-flash | 14 | 11 | 1/1; 17/17 | 3152 / 17557 | $0.000142620 / $0.000778050 |
| contractnli-test-52 | alibaba/qwen3.7-flash | 14 | 12 | 1/1; 17/17 | 3089 / 23550 | $0.000185320 / $0.001494980 |

The Jev bundle and one separate call on contract 52 failed in the first pass. Their missing cost records prevent a complete cost comparison. Separate time sums sequential calls per model; it is not a parallel fan-out measurement.

## Caching and routing

BANKING77: Luna reported cache hits on 76/77 calls (93,936 cached tokens of 96,622 prompt tokens); Qwen on 74/77 (94,720 of 104,268). Jev usage did not expose comparable cache counts. Do not interpret unavailable cache data as zero caching. Gateway made more than one provider attempt on some Jev requests; see routing metadata. All successful Jev calls resolved to TypeSafe, Luna to OpenAI and Qwen to Alibaba.

## Service failures and recovery

Nine first-pass Jev requests returned HTTP 503. A single later pass repeated only those exact requests. Seven succeeded; contract bundles 2 and 542 returned 503 again. The original records are unchanged. This is a recovery diagnostic, not a benchmark of a configured production retry/backoff policy. After recovery, Jev delivered 77 banking responses with 61 label matches and 10 contract bundles with 128/170 label matches.

## Inspectable errors

BANKING77 test row 1254 asks “what is my card PIN” but is labeled `get_physical_card`. All three predicted `change_pin`. The author would review this annotation; the recorded score retains it. Other confusions include transfer timing versus a missing transfer and neighboring physical-card intents.

## Run files

- [Summary, per-class results and all disagreements](runs/2026-09-25T17-07-27.378Z/summary.json).
- [First-pass manifest](runs/2026-09-25T17-07-27.378Z/manifest.json) and [request/response records](runs/2026-09-25T17-07-27.378Z/results.jsonl).
- [Recovery manifest](runs/2026-09-25T17-07-27.378Z/recovery/manifest.json) and [records](runs/2026-09-25T17-07-27.378Z/recovery/results.jsonl).
- [Frozen fixtures and source hashes](cases.json).
- [Model catalog at run time](runs/2026-09-25T17-07-27.378Z/model-catalog.json).