Evaluation methods.
Original evidence.

Use Jev for a hosted pilot.

Pin jev-1.13.0 to preserve the hosted behavior while adopting the Sys1 client. sys1 jev enable selects hosted-only routing. A provider outage returns an error; it does not silently substitute Qwen.

Evaluate Qwen for local use.

Keep requests on your machine after model setup. Local probability readout is experimental and is not interchangeable with Jev. Select a model explicitly and check its decisions on representative cases before using them to drive actions.

CUA and Needle adapters are removed in Sys1 0.9. Their original results remain available. OpenJev can be registered as an explicitly selected HTTP server; its extensions and server lifecycle remain separate. Compatible server guide.

Jev timing includes the network round trip, provider processing, and the full response body. Local timing excludes HTTP. These boundaries do not measure isolated model speed. Provider hardware, caching, and client region are unverified.

The broader hosted run is incomplete. Authentication was rejected with HTTP 401; the harness stopped after three attempts, with no valid answer. It supplies no new quality or speed result. The completed forms-v1 report remains separate.

What this means: djev is a hosted Maisa DiffusionGemma implementation; its 74.3 is not a local DiffusionGemma result. The OpenJev DiffusionGemma row is a separate GPU server run. OpenJev can be registered as an operator-owned Sys1 HTTP backend; this external run does not qualify that deployment or establish Mac-local quality. SemIf’s Qwen3.5 4B uses a different adapter and BF16 weights, so its score does not apply to Sys1’s bundled Qwen GGUF path.

The suite freezes 220 hard decisions (111 public, 109 held out) and 534 decisions per system. Its 25:25:25:25 geometric score combines intelligence, calibration, serial p50/p95 speed, and dollars per 1,000 decisions. These headline values are point estimates from one serial run; they are not confidence intervals or a seed sweep. Evaluator-owned server latency receives a ×2 + 0.15 s adjustment; public demo latency receives ×2. These are assumptions, not measurements. The hard set is English and authored/reviewed by language models, and held-out cases are sent to evaluated services, so this is useful external evidence rather than a contamination-proof production benchmark.

Scroll tables sideways to see all columns.

Fast, with a narrow quality result. 30/72 correct (41.7%) is only three cases above the constant-label reference. Of the 20 order-stable choices, nine were always correct and eleven always wrong. These results do not support replacing Jev or promoting Laya as Sys1's local default.

Latency measures Agent.predict, including tokenization and synchronized inference, but excludes IPC, validation, loading, and HTTP. Throughput includes IPC and validation. Qwen's local-runner latency includes IPC, so these columns are not an isolated model-speed comparison. MLX 0.32.2 · Python 3.12.14 · GPU · batch 1 · compilation and prompt caching disabled.

Scroll tables sideways to see all columns.

Experimental. 44/72 correct (61.1%) is not dependable general decision quality. A constant-label reference scores 27/72. Of the 13 order-stable choices, ten are always correct and three always wrong.

Apple M5 Max · 36 GiB system memory · Darwin 25.5.0 · Bun 1.3.14 · Q4_K_M · Metal · 2,048-token context. Shared host, with no measured peak memory or GPU utilization. Three passes returned identical decisions; repetitions add timing observations, not new quality cases.

Scroll tables sideways to see all columns.

The best fixed label in each choice family, a constant yes/no answer, and a constant score level together get 27/72. Qwen3 1.7B gets only five more cases right. These are authored synthetic workflows, not a representative production sample.

Apple M5 Max · 36 GiB system memory · macOS/Darwin 25.5.0 · Bun 1.3.14 · Metal capability probe. Shared host; GPU utilization and peak model memory were not measured. Timing includes the local runner and worker communication, excludes HTTP, and is separate from correctness.

Local Qwen

Counts the actual wrapped prompt for each question. Output usage is zero because Sys1 reads first-token probabilities without generating answer text. This still uses compute, memory, and energy; it is not free inference.

Hosted Jev

The completed 100-call forms-v1 phase reported 34,760 input and 3,800 output tokens, with estimated input cost of $0.00145992. All 105 calls cost an estimated $0.00153342. Output is free at the published rate. These are usage-based estimates, not invoices.

1. Quality: 72 authored cases, 24 per answer type. Grade only the first pass. Choice requires the exact label; yes/no uses a strict 0.5 threshold; score uses a unique most-probable level. Noul and Score ties, and all failures, count as incorrect.
2. Stability: all six option orders for 24 choice cases. Report identical semantic answers separately from correctness; permutations do not add independent cases.
3. Timing: three measured passes, 216 calls at concurrency one. Two earlier form cases warm up the route. p50/p95 use nearest rank over all attempts; throughput includes response validation.
4. Boundaries: pin weights and the provider model version; keep local and network timing distinct. A small project-authored test does not establish production quality or calibrated probabilities.