Choose your model.
Know the evidence.
Sys1 keeps your application on one typed interface. Choose the backend using measured correctness, latency, token use, and failure behavior. Hosted Jev and experimental local Qwen have different limits.
Measured September 19–20, 2026 · Full results, including wrong answers, available below
Two paths. An explicit choice.
Keep your application's questions, permissions, and decision policy. Let Sys1 own transport, response validation, model selection, and local model lifecycle.
- Use Jev for a hosted pilot.
- Pin
jev-1.13.0to preserve the hosted behavior while adopting the Sys1 client.sys1 jev enableselects hosted-only routing. A provider outage returns an error; it does not silently substitute Qwen. - Evaluate Qwen for local use.
- Keep requests on your machine after model setup. Local probability readout is experimental and is not interchangeable with Jev. Select a model explicitly and check its decisions on representative cases before using them to drive actions.
CUA and Needle adapters are removed in Sys1 0.9. Their original results remain available. OpenJev can be registered as an explicitly selected HTTP server; its extensions and server lifecycle remain separate. Compatible server guide.
Jev is measured, too.
The completed hosted run uses the earlier forms-v1 test. Its 20/20 result is encouraging for that small task; it is not a score on the broader 72-case evaluation.
| Route | Correct cases | p50 | p95 | Valid responses |
|---|---|---|---|---|
| Jev 1.13.0 · HTTP | 20/20 | 246.1 ms | 343.0 ms | 100/100 |
| Qwen3 1.7B · local adapter | 18/20 | 258.3 ms | 271.4 ms | 100/100 |
| Qwen3 0.6B · local adapter | 8/20 | 206.6 ms | 225.1 ms | 100/100 |
Jev timing includes the network round trip, provider processing, and the full response body. Local timing excludes HTTP. These boundaries do not measure isolated model speed. Provider hardware, caching, and client region are unverified.
For cross-model context: the pinned JevBench v1.2.6 scorecard compares Jev-class systems with a separate 534-decision methodology. Its external scores do not qualify a Sys1 adapter or replace the measurements on this page.
The broader hosted run is incomplete. Authentication was rejected with HTTP 401; the harness stopped after three attempts, with no valid answer. It supplies no new quality or speed result. The completed forms-v1 report remains separate.
Laya: a different local approach.
Laya MLX uses trained heads for choices, probabilities, and scores. We tested its typed-decisions checkpoint directly on the same decisions-v3 cases as Qwen3.5. This is a candidate study; Laya is not bundled into Sys1.
Scroll tables sideways to see all columns.
| Model | Correct | Choice | Yes / no | Score | Order-stable cases |
|---|---|---|---|---|---|
| Laya typed-decisions | 30/72 | 9/24 | 13/24 | 8/24 | 20/24 |
Fast, with a narrow quality result. 30/72 correct (41.7%) is only three cases above the constant-label reference. Of the 20 order-stable choices, nine were always correct and eleven always wrong. These results do not support replacing Jev or promoting Laya as Sys1's local default.
| Native p50 | Native p95 | Valid decisions / s | Valid responses | Input / output tokens | Weights |
|---|---|---|---|---|---|
| 9.7 ms | 15.3 ms | 89.62 | 216/216 | 29,157 / 0 | 803.6 MiB |
Latency measures Agent.predict, including tokenization and synchronized inference, but excludes IPC, validation, loading, and HTTP. Throughput includes IPC and validation. Qwen's local-runner latency includes IPC, so these columns are not an isolated model-speed comparison. MLX 0.32.2 · Python 3.12.14 · GPU · batch 1 · compilation and prompt caching disabled.
What integration would require
Laya's 1,024-token context has a separate 256-token question budget. Upstream can silently shorten instructions, options, and state. Our preflight checked every complete token sequence and option marker: all 362 calls fit unchanged, with at most 166 tokens. A future runner must reject overflow rather than discard evidence.
The raw API uses a generic model name and supplies extra action fields. The study preserves that response alongside an explicitly normalized answer with the pinned checkpoint identity. A production integration also needs bounded request handling and process-owned cancellation. Sys1's external-backend interface is a possible connection point; this adapter is not shipped.
The publisher's fidelity tests measure agreement with its upstream runtime. Its separate AG News sample covers one classification task in the training mix. Neither establishes general quality on the workflows measured here. Upstream benchmarks and limitations.
Qwen remains an experiment.
Qwen3.5 4B is the newest evaluated Sys1 candidate. Its 72 cases cover choices, yes/no conditions, and rubric scores, including boundary cases. They were frozen before the model ran.
Scroll tables sideways to see all columns.
| Model | Correct | Choice | Yes / no | Score | Order-stable cases |
|---|---|---|---|---|---|
| Qwen3.5 4B | 44/72 | 16/24 | 16/24 | 12/24 | 13/24 |
Experimental. 44/72 correct (61.1%) is not dependable general decision quality. A constant-label reference scores 27/72. Of the 13 order-stable choices, ten are always correct and three always wrong.
| p50 | p95 | Valid decisions / s | Valid responses | Input / output tokens | Weights |
|---|---|---|---|---|---|
| 464.7 ms | 540.8 ms | 2.13 | 216/216 | 44,604 / 0 | 2.55 GiB |
Apple M5 Max · 36 GiB system memory · Darwin 25.5.0 · Bun 1.3.14 · Q4_K_M · Metal · 2,048-token context. Shared host, with no measured peak memory or GPU utilization. Three passes returned identical decisions; repetitions add timing observations, not new quality cases.
Failure patterns and the reasoning experiment
Qwen3.5 missed eight of twelve false propositions and favored score level 1 on thirteen of twenty-four cases. This measures the Sys1 first-token adapter, not Qwen's full generative capabilities.
A separate, fixed twelve-case feasibility test allowed 512 reasoning tokens before reading the answer. Nine attempts reached the cap. Correct answers fell from 7/12 to 3/12; median attempt time rose from 495 ms to 9,867 ms. This previously observed subset is exploratory, and the mode was not adopted. Complete feasibility report.
The earlier Qwen results stay public.
The 72-case decisions-v2 test exposed limits that the initial form-action test missed. Its cases differ from v3 above, so scores across the two fixtures do not measure an improvement.
Scroll tables sideways to see all columns.
| Model | Correct | Choice | Yes / no | Score | Order-stable cases |
|---|---|---|---|---|---|
| Qwen3 1.7B | 32/72 | 12/24 | 13/24 | 7/24 | 7/24 |
| Qwen3 0.6B | 25/72 | 8/24 | 12/24 | 5/24 | 0/24 |
The best fixed label in each choice family, a constant yes/no answer, and a constant score level together get 27/72. Qwen3 1.7B gets only five more cases right. These are authored synthetic workflows, not a representative production sample.
What failed, and why confidence does not solve it
Qwen3 1.7B answered “yes” on 23/24 yes/no cases. It chose score level 1 on 20/24 score cases and never chose levels 0 or 3. Of its 40 wrong first-pass answers, 35 had reported confidence of at least 0.90.
We also tried every option order for each choice case. Qwen3 1.7B kept the same semantic answer on just 7/24 cases; two of those stable answers were always wrong. Qwen3 0.6B was stable on none. Order stability is a diagnostic, not another accuracy score.
All three measured passes returned identical decisions. Repetition made the behavior reproducible; it did not make it correct. Sys1 does not treat local confidence as permission to take an action.
| Model | p50 | p95 | Valid / second | Valid responses | Input / output tokens |
|---|---|---|---|---|---|
| Qwen3 1.7B | 213.6 ms | 247.6 ms | 4.63 | 216/216 | 45,183 / 0 |
| Qwen3 0.6B | 209.6 ms | 255.0 ms | 4.73 | 216/216 | 45,183 / 0 |
Apple M5 Max · 36 GiB system memory · macOS/Darwin 25.5.0 · Bun 1.3.14 · Metal capability probe. Shared host; GPU utilization and peak model memory were not measured. Timing includes the local runner and worker communication, excludes HTTP, and is separate from correctness.
Count decisions, tokens, and cost.
Use correctness and completed decisions alongside latency. Token counters describe different inference paths; they are not a shared measure of model work.
- Local Qwen
- Counts the actual wrapped prompt for each question. Output usage is zero because Sys1 reads first-token probabilities without generating answer text. This still uses compute, memory, and energy; it is not free inference.
- Hosted Jev
- The completed 100-call forms-v1 phase reported 34,760 input and 3,800 output tokens, with estimated input cost of $0.00145992. All 105 calls cost an estimated $0.00153342. Output is free at the published rate. These are usage-based estimates, not invoices.
Estimate Jev input cost.
Published price for Jev 1.13: $0.042 per million input tokens, with output free. Change request volume and average billed input tokens to estimate a monthly total.
Arithmetic only. Excludes retries and any future price changes. Local models have hardware, memory, energy, and operating costs instead of this API charge.
Use the provider’s usage counters, not JSON byte length.
Evidence you can rerun.
The fixtures, grading rules, model pins, source hashes, and complete responses are public. No case is removed for producing an inconvenient answer.
- Quality: 72 authored cases, 24 per answer type. Grade only the first pass. Choice requires the exact label; yes/no uses a strict 0.5 threshold; score uses a unique most-probable level. Noul and Score ties, and all failures, count as incorrect.
- Stability: all six option orders for 24 choice cases. Report identical semantic answers separately from correctness; permutations do not add independent cases.
- Timing: three measured passes, 216 calls at concurrency one. Two earlier form cases warm up the route. p50/p95 use nearest rank over all attempts; throughput includes response validation.
- Boundaries: pin weights and the provider model version; keep local and network timing distinct. A small project-authored test does not establish production quality or calibrated probabilities.
Related projects and publisher benchmarks
Jev is a hosted decision model; Qwen supplies open model weights; llama.cpp supplies local inference. Sys1 connects these to one typed application contract. OpenJev offers a separate compatible server you can operate and explicitly register.
Publisher speed and quality results use different workloads, runtimes, and hardware. They remain in the upstream evidence archive, with primary sources and conditions. They do not form a shared leaderboard or validate Sys1's local adapter.