We Put Two Jev-Inspired Local AI Models in Front of Risky Software Actions

What happens if an AI model does not write prose, but instead sits between a proposed software action and the code that may execute it?

We built that experiment on a MacBook with two open local decision systems: SemIf on Qwen3.5-4B and kev-0.5b. We gave both models the same structured state, asked the same typed questions, measured accuracy and probability quality, isolated the evaluator in Docker, and then turned the result into a Next.js demonstration called AI Bouncer.

The short answer is useful but not magical: SemIf was substantially more accurate and better calibrated on our fixed corpus; Kev was dramatically faster. The most important design choice was not choosing a winner. It was keeping both models behind a deterministic policy layer that can allow, escalate, or block an action, but never execute it.

Result: SemIf scored 86.1% accuracy; Kev scored 65.8%. Kev answered an average request in 78.3 ms, versus 1,365.7 ms for SemIf. Those are benchmark aggregates, not the probability attached to any one live decision.
Public repository: Decision Model Lab contains the benchmark harness, fixed corpus, tests, Docker setup, and AI Bouncer source.1

Jev is the reason this experiment exists

TypeSafe announced Jev on September 15, 2026 as its first System One Model: a model class designed to turn unstructured state into fast, typed probabilistic decisions that software can use directly. That announcement is the catalyst for this entire build, especially its claim that AI needs an interface software can depend on.2

But we did not benchmark Jev. There is no Jev result in our 86-case corpus. We tested SemIf and Kev because they were open implementations we could inspect, run locally on the MacBook, and place behind the same Jev-like /v1/systemone contract.

That distinction matters:

So the 86.1% and 65.8% figures compare SemIf with Kev on our own fixed suite. They are not Jev scores, they do not reproduce TypeSafe’s evaluation, and they should not be read as a claim that either open model is equivalent to Jev.

The interface these models actually change

This is not simply “AI versus software,” and it is not really “human versus software” either.

The interesting interface is between unstructured state and deterministic application logic. Normal software wants booleans, enums, scores, and explicit rules. A language model usually wants to generate an explanation that the application must parse. Decision models compress that gap: give them shared state plus typed questions, and they return option probabilities.

SemIf’s primary repository describes a direct scorer that returns typed option probabilities without an answer sentence or decoding loop, and it supports a native MLX backend on Apple silicon.3

The kev-0.5b model card describes a LoRA adapter and pointer head on Qwen2.5-0.5B that returns distributions over typed questions in one forward pass; it also labels the checkpoint a research prototype rather than a production model.4

That still does not make the model the policy. A probability such as 0.92 means “this model placed 92% of its distribution here,” not “the action is 92% safe.” The application must decide how evidence becomes policy, when disagreement triggers review, and which states are forbidden regardless of confidence.

Building a fixed 86-case benchmark

Every backend receives the same contract:

The corpus contains 86 cases and 202 scored decisions. The cases cover routing, authorization, data sensitivity, reversibility, prompt injection, and action risk. We froze the corpus and recorded its digest before comparison so a later model could not quietly receive an easier suite.

Both runs completed all 86 requested cases, produced all 202 valid decisions, and recorded zero backend errors. That matters: an impressive score with silently dropped cases would not be comparable.

We also kept a deterministic oracle backend, but only to test the harness. Its results are deliberately excluded from model rankings. The oracle proves that scoring, reporting, and replay work; it says nothing about model quality.

What 86.1% accuracy means and what it does not

SemIf selected the expected label on 174 of 202 decisions, which is 86.1%. Kev selected the expected label on 133 of 202, or 65.8%.

Accuracy is calculated over the fixed labelled corpus:

correct decisions ÷ all scored decisions

It is not the mean confidence, and it is not a promise that the next action will be classified correctly.

Probability quality needs its own measurements. We recorded:

| Model | Accuracy | Mean latency | P50 | P95 | Brier score | ECE |

|---|---:|---:|---:|---:|---:|---:|

| SemIf Qwen3.5-4B MLX | 86.1% | 1,365.7 ms | 1,280.3 ms | 1,894.8 ms | 0.186 | 0.060 |

| Kev 0.5B | 65.8% | 78.3 ms | 72.8 ms | 167.0 ms | 0.456 | 0.115 |

A Brier score measures squared error in the probabilities; lower is better. Expected calibration error (ECE) checks whether answers assigned roughly 80% confidence are actually right about 80% of the time; again, lower is better. On this corpus, SemIf was not only more often correct; it also produced probabilities that were more useful as evidence.

Benchmark bars compare SemIf and Kev accuracy, mean latency, and Brier probability score across 86 cases and 202 decisions.

The latency result points the other way. Kev was approximately 17.4 times faster by mean request latency. If the decision sits in a high-frequency, low-consequence route, that tradeoff may matter more than the aggregate accuracy gap. If the decision protects a destructive action, a second of latency may be cheap.

The honest conclusion is not “SemIf wins” or “Kev wins.” It is that model choice belongs to a measured operating point: task distribution, error cost, latency budget, and calibration threshold.

Docker for isolation, native macOS for inference

We wanted Docker because the evaluator, web app, dependencies, and report generation should be reproducible. We did not put the two model processes in ordinary Linux containers on the Mac.

Docker’s own GPU page says ordinary Docker Desktop GPU support is only available on Windows with the WSL2 backend.5

Current Docker Model Runner can use Metal on Apple silicon, but our tested upstreams already exposed a custom /v1/systemone contract through their own MLX and MPS launchers. Moving them into a different runtime would have changed the thing we were benchmarking.

So we split the boundary:

A Mac runs two local decision models beside an isolated Docker evaluator, connected without placing Metal inference inside the container.

This also answered another practical question: we did not need Ollama. Ollama is excellent when the model already fits its serving contract. Here, both projects already had typed decision endpoints, model-specific token handling, and working Apple-silicon paths. Adding another server would have added translation without improving the experiment.

Turning the benchmark into AI Bouncer

A benchmark report is useful to engineers. It is not a compelling demonstration of why this model category matters.

AI Bouncer makes the decision boundary visible. The user chooses a proposed action, such as sending data, deleting a resource, changing access, or following suspicious instructions, and watches five gates evaluate it:

  1. Authorization: is the actor permitted?
  2. Sensitivity: what class of data is involved?
  3. Reversibility: can the action be undone?
  4. Injection: does the state contain instruction-hijacking signals?
  5. Risk: what is the likely consequence?

Kev and SemIf produce their answers independently. The interface displays the selected options, probabilities, latency, and disagreement. A separate resolver then produces one of three outcomes:

AI Bouncer dashboard comparing Kev and SemIf across authorization, sensitivity, reversibility, injection, and risk gates.

The resolver is conservative on purpose. It does not average away a severe warning, and it does not treat a missing model as a silent approval. Most importantly, the demo never executes the represented action. It evaluates a proposal and exposes the evidence so a person can inspect the boundary.

Replay and live inference use the same normalized envelope. That lets us record a reliable demonstration without pretending replay is live, then switch to Kev, SemIf, or both when the services are available.

Start and stop controls without pretending process management is simple

The first version assumed the model servers were already running. That made the demo feel broken even when the application was healthy, so we added explicit lifecycle controls.

The UI now shows each model as stopped, starting, ready, stopping, or unavailable. A Start button launches the repository’s known script. A Stop button only terminates a process the app itself started and still owns. It will not hunt for an arbitrary PID on the machine.

That constraint is less flashy than a global “kill server” command, but it is a better demonstration of the same principle as the model gate: represent authority explicitly, fail closed, and do not let a convenient interface exceed its mandate.

The lifecycle routes also validate model identity and health before reporting readiness. A listening port alone is not enough; the service must answer as the expected model.

What the experiment proves

This build demonstrates that:

It also produced a useful content lesson: the most interesting screen is not a chatbot. It is the moment two models disagree over a real action and the software refuses to blur that disagreement into one confident answer.

What it does not prove

This is a research demonstration, not a production safety certification.

The corpus is intentionally broad but still small. Its labels reflect our written policy. Neither model was tested against every domain, language, adversarial input, or distribution shift. The probabilities should not be treated as universal truth, and the benchmark cannot establish how either system behaves inside a different application.

Before using this pattern for consequential decisions, we would add domain-specific labelled outcomes, calibration by slice, explicit abstention thresholds, adversarial tests, audit logs, version pinning, and shadow-mode observation. Any irreversible or high-impact action would still retain a human escalation path.

That caveat is not a weakness of the demonstration. It is the point: a useful AI interface makes uncertainty and authority visible enough for software and people to act responsibly.

By the numbers

1 verified source Checked in the AlkenaCode editorial review