# Pilot results — September 24, 2026

216 successful scored calls: 18 synthetic cases × two criterion orders × six models. Every response supplied valid JSON/typed output with a permitted label. Each model completed 36 judgments; these are paired variants of 18 cases, not 36 independent examples. See method below.

| Model | Correct judgments | Median end-to-end latency | Market cost, 36 calls | Reported charge, 36 calls | Agreement across 18 variant pairs |
|---|---:|---:|---:|---:|---:|
| Jev | 35/36 | 314 ms | $0.00064940 | $0.00000000 | 17/18 |
| GPT-6 Luna | 36/36 | 1223.5 ms | $0.00083880 | $0.00083880 | 18/18 |
| GPT-OSS-20B | 31/36 | 1127.5 ms | $0.00043508 | $0.00043508 | 17/18 |
| GPT-OSS-120B | 34/36 | 692.5 ms | $0.00173730 | $0.00173730 | 18/18 |
| Qwen 3.7 Flash | 34/36 | 894.5 ms | $0.00029047 | $0.00029047 | 18/18 |
| DeepSeek V4 Flash 0731 | 34/36 | 1795 ms | $0.00138101 | $0.00138101 | 17/18 |

Total Gateway-reported market cost: **$0.00533206**. Reported account charge: **$0.00468266** (about 0.47 US cents), excluding the setup smoke calls. Jev's zero reported charge is kept separate from its market cost.

## What this supports

- Luna was a credible cheap baseline: 36/36, with about 1.29× Jev's market cost and 3.90× its median application latency. This is a narrow pilot observation, not evidence that Luna is universally more accurate.
- Jev had the lowest median latency here and returned a native probability distribution in addition to a decision. Those are concrete reasons to evaluate it, even when a cheap language model can return the required label.
- Qwen had the lowest observed market cost, about 45% of Jev's, with 34/36 correct judgments. GPT-OSS-20B was also cheaper than Jev, but scored 31/36.
- All mistakes occurred in routing; every model scored 12/12 in both grounding and the policy family. These easy families hit the ceiling and do not distinguish capability well.
- The comparison is shaped by prompt wrappers, tokenization, reasoning, caching, serving provider, and network path. It does not isolate the model architecture.

## Inspectable errors

- Jev misclassified the injection-bearing invoice ticket (`route-6`) only with reversed criteria, returning mixed instead of billing. It assigned mixed 0.49, billing 0.45, and confidence 0.32. That uncertainty signal is useful to inspect; one error cannot establish calibration or validate a threshold.
- Luna made no errors on these fixtures.
- GPT-OSS-20B counted a resolved payment problem as still active in both orders, inferred an access issue from a vague account complaint in both orders, and missed the injection-bearing ticket in one order.
- GPT-OSS-120B and Qwen both counted the resolved payment problem as still active in both orders. Stable answers can still be wrong.
- DeepSeek missed the injection-bearing ticket in both orders, returning different wrong labels: other and access. This single case does not quantify prompt-injection resistance.

## Serving conditions

| Requested model | Final provider(s) |
|---|---|
| Jev | typesafe-ai: 36 |
| GPT-6 Luna | openai: 36 |
| GPT-OSS-20B | deepinfra: 36 |
| GPT-OSS-120B | baseten: 36 |
| Qwen 3.7 Flash | alibaba: 36 |
| DeepSeek V4 Flash 0731 | alibaba: 26, baseten: 10 |

DeepSeek had 26 calls with more than one provider attempt: Baseten returned HTTP 503 (service temporarily unavailable), after which Alibaba served the request. Their full end-to-end latency remains included. No application retries or cross-model fallback were requested. The Chinese models ran in a second batch after the first four models, so cross-batch latency comparisons are directional.

GPT-OSS-20B reported 1,176 reasoning tokens; GPT-OSS-120B reported 1,079 reasoning tokens and 3,616 cached input tokens. Luna, Qwen, and DeepSeek reported zero reasoning tokens. Jev reported 15,462 input tokens across 36 calls versus Luna's 6,188, illustrating why advertised input-token price alone does not predict cost per decision.

Gateway records confirm the requested model slugs and serving providers, but do not independently verify underlying weight revisions for aliases. Do not claim a pinned Jev revision from this run.


---

# Jev versus inexpensive language models: diagnostic pilot

Run date: September 24, 2026. This is a small, agent-authored synthetic pilot, not a representative benchmark or a production safety evaluation.

## Question

When an application needs one short decision, does Jev improve enough on an inexpensive language model with structured output to justify a separate integration? Assess observed correctness, application latency, and cost separately. Do not assume the launch's comparisons against more elaborate LLM workflows apply to this workload.

## Method

- Eighteen fixtures, six each for ticket routing, evidence-grounded classification, and interpreting a refund policy. Expected labels were saved before inference; no gold label was sent to a model. There was no independent human adjudication.
- Every case ran twice: original criterion order and reversed criterion order. These are **18 unique cases and 36 judgments per model**, not 36 independent examples. Each variant ran once; differences can reflect sampling as well as ordering.
- Same state, instructions, criteria, and labels across models, using each API's native input shape. Jev received a native choice question; chat models received system instructions and a strict JSON schema containing only `label`. No explanation or verbal confidence requested.
- Jev used `POST /v1/evaluate`; other models used `POST /v1/chat/completions`. Both used Vercel AI Gateway. This compares application paths, not isolated architecture or hardware speed.
- GPT-6 Luna, Qwen 3.7 Flash, and DeepSeek V4 Flash 0731 requested reasoning `none`. GPT-OSS models requested their lowest advertised setting, `low`. Reasoning token use and differences in token accounting are retained in the logs. No explicit temperature or seed was set.
- Requests were sequential, with model order rotated between cases. The initial four-model batch completed before the two Chinese models ran as an additional batch. Thus the Chinese results share fixtures but not the same time window or interleaving as Jev. Latency includes network, Gateway, provider processing, and complete response transfer from this local client. No warm-up calls were dropped from the pilot.
- Gateway chose providers dynamically. Logs preserve provider identity, attempts, cache use, generation ID, finish reason, and cost. No cross-model fallback was requested. No application-level retries; timeout 45 seconds; chat completion cap 768 tokens. A model restriction would be recorded and then skipped.
- Early smoke calls are excluded. Luna initially returned a free-tier restriction during setup, but the user added credits before the scored run; its scored batch completed normally.
- All data are synthetic. No customer information or real refund execution was involved. For the fully structured policy fixtures, ordinary deterministic code is the appropriate production baseline; a model is unnecessary once the facts are established.

## Accounting

Use Gateway `marketCost` for model comparisons and keep `cost` as the reported account charge. Jev returned a zero charge but a nonzero market cost on this run; zero charge is not evidence of permanently free inference. Provider pricing, caching, input wrappers, and reasoning tokens can change the actual cost from a simple advertised input-token comparison.

The model-catalog snapshot is in [gateway-models.json](gateway-models.json). Published short-context prices per million input/output tokens at inspection were Jev $0.042/$0, Luna $0.10/$0.50, GPT-OSS-20B $0.03/$0.14, GPT-OSS-120B $0.10/$0.50, Qwen 3.7 Flash $0.03/$0.13, and DeepSeek V4 Flash 0731 from $0.076/$0.153. Some vary by provider; use the actual run metadata for the comparison. Qwen is included as an inexpensive Chinese API model; this report does not claim its weights are available. DeepSeek's dated model has a public weight release.

## Limits and next experiment

These short examples have few errors and many ceiling scores. They do not establish population accuracy, robust prompt-injection resistance, calibration, production throughput, long-context performance, or a universal winner. No uncertainty threshold was tuned or validated. Reversed-order agreement includes consistently wrong answers.

A follow-up should freeze a human-reviewed held-out set from the intended workload, test realistic context lengths and multiple questions over shared state, repeat requests at controlled concurrency, and compare error/review cost at a target latency. A shared-state multi-question test matters because it exercises an explicit Jev feature absent from this single-choice pilot. Choose any abstention threshold on a separate development set and evaluate it without retuning on the holdout.

## Sources

- [Vercel: evaluation endpoint and typed questions](https://vercel.com/docs/ai-gateway/modalities/evaluation).
- [TypeSafe: Jev model specifications and prices](https://docs.typesafe.ai/models).
- [TypeSafe: launch and vendor comparisons](https://typesafe.ai/blog/introducing-system-one-models-and-jev), September 15, 2026.
- [TypeSafe: confidence and probabilities](https://docs.typesafe.ai/confidence).
- [OpenAI: GPT-6 Luna price and supported settings](https://developers.openai.com/api/docs/models/gpt-6-luna).
- [OpenAI: structured outputs](https://developers.openai.com/api/docs/guides/structured-outputs).
- [Vercel: current model catalog](https://ai-gateway.vercel.sh/v1/models).
- [Vercel: Qwen 3.7 Flash](https://vercel.com/ai-gateway/models/qwen3.7-flash).
- [Vercel: DeepSeek V4 Flash 0731](https://vercel.com/ai-gateway/models/deepseek-v4-flash-0731).
- [DeepSeek: public model card and weights](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731).
- [Zhang et al.: Same Scores, Different Decisions](https://arxiv.org/abs/2609.27678), September 23, 2026 preprint. Its ContractNLI findings provide context, not validation of this pilot.

## Reproduction files

[Download reproduction bundle](reproduction.zip). [Observations](observations.json), [fixtures](cases.json), [summary](combined-summary.json), [model catalog](gateway-models.json). Operational request identifiers and client fingerprints are omitted. The prompts, answers, token counts, provider names, attempts, costs and timing remain inspectable.