The original
form-action comparison.

Scroll tables sideways on smaller screens to see all columns.

MiB = 1,048,576 bytes. File sizes are verified download footprints, not RAM requirements. Sys1 caches one local model by default; that limits model count, not memory bytes. Registry & artifact pins.

Apple M5 Max · 36 GiB system memory · macOS / Darwin 25.5.0 · Bun 1.3.14. The native capability probe selected Metal; per-inference GPU usage was not instrumented. Local adapters ran on this shared development machine on September 19; hosted Jev ran from the same host on September 20, 2026. These are separate snapshots.

The timing boundaries differ. Local latency measures the adapter and excludes HTTP. Jev latency includes the network round trip, provider processing, and complete response body. Neither is a controlled comparison of inference speed.

The smaller local routes are not reliable enough for this task. CUA, Needle, and Qwen 0.6B each reached 8/20, barely above the 7/20 majority-label baseline. Qwen 1.7B missed two cases; Jev answered all 20 correctly. Start an evaluation of similar decisions with Jev, or Qwen 1.7B when local execution is required. Twenty authored cases still do not qualify either for unattended production use.

• These are whole-form narrative questions: submit, correct, or wait. CUA’s native training format uses individual TASK/FORM/ELEMENT observations and fill/check/click/skip actions. Its score here measures transfer to this request format, not native form-filling quality. Needle is also being adapted from extraction into a choice task.
• Qwen and CUA reuse a resident model after two warmups. Needle starts a new process on every call. The latency boundary is the local adapter, including model work and IPC, excluding HTTP/gateway routing, downloads, and response validation. Throughput includes loop and validation overhead.
• Repeating the same 20 cases five times gives 100 timing observations, not 100 independent quality examples. All 500 repeated calls across the five routes returned valid responses, with no errors or timeouts. A valid response can still be wrong. OpenJev was not measured.
• Jev used the pinned jev-1.13.0 model with no retries: three initial client calls, two warmups, then 100 measured calls. Client region is unverified; provider hardware, load, and caching are unknown. One question per request does not test Jev’s shared-state, multi-question advantage.

Local source: 83ca299; Jev source: e770483. Both runs used the identical fixture, runtime source, and lockfile, with relevant files unmodified. Raw reports record full provenance. Local weights have verified hashes; Jev’s version is provider-reported. Percentiles use nearest rank. The reports preserve the original measurements without reranking or removing failed cases.

Qwen through Sys1

Reports the actual tokenized prompt count, summed across questions. Each question evaluates its own wrapped prompt, so repeating a large state can cost input processing time. Output usage is zero because Sys1 reads the first-token probability distribution instead of returning generated text.

CUA-S1 Forms

Scores UTF-8 byte representations directly. Generative tokens per second is not an applicable metric. The wire response uses zero token counters.

Cactus Needle

Performs native tool-call inference. Sys1 currently returns zero token counters because this adapter does not expose accounting. Read them as unreported, not as zero model work.

Hosted Jev

Reads shared state once and evaluates questions in parallel. In our 100-call run it reported 34,760 input and 3,800 output tokens; output is free at the published price. Token counts differ from Qwen because the models and request processing differ. Its documented account rate limits describe allowed traffic, not achieved inference speed. Current model & pricing.

1. Check correctness and failures. A valid answer shape can still contain a wrong decision. Keep wrong answers, timeouts, and unsupported requests visible.
2. Separate first call from steady use. Loading weights and starting processes can dominate a short request. OS file caching makes a fresh runner different from a cold machine.
3. Keep workload size explicit. Record state length, question count, options, concurrency, and context configuration. Report p50 and p95 latency with sample counts.
4. Evaluate confidence separately. A normalized probability or model-reported confidence is not evidence of calibration. Test thresholds against held-out labels before letting them authorize actions.

Sys1’s advantage is that you can run this comparison behind the same application contract, then choose a policy or an explicit model pin. Switching the interface does not establish interchangeable model quality.