Twelve Local Models, One Harness That Checks Them

In short. Twelve open-weight models, from 3B to 24B parameters, went through the qualification protocol of Talos, a local coding assistant, on one desktop. The six that completed a 72-trial run passed every trust gate in every trial, a joint result of the model and the runtime checks around it, and did the work unevenly. Forty-three of the 102 failed trials were results affected by runtime or scoring behavior. Those results cannot be treated as clean measurements of model quality, so the protocol measured Talos alongside the models. These are descriptive results from one machine and one engine build, not a ranking.

Talos runs a local model through llama.cpp, and every tool call the model makes is a proposal the runtime checks before it takes effect. Knowing whether a model calls tools when it should, abstains when it should and finishes the work it starts takes a protocol. The results below come from the packets it left behind, scorecards, per-trial results, final files and traces, and from the reports written beside them.

The protocol

Each candidate passes through fixed stages. A failed stage stops that configuration, and a material change, such as a new request layout, starts a new configuration identity.

1. Acquisition. The exact source revision, bytes, size, SHA-256 and license are frozen. Where an official repository did not publish the needed quantization, a local conversion was made twice, and both outputs were byte-identical.
2. Engine preflight. The pinned llama.cpp server answers short direct probes, a request with no tools and one that must call a tool, with server-default sampling and output caps of 64 and 256 tokens. Some candidates were probed with a single user message, others with two short system messages before it. These probes are simpler than the requests Talos really sends.
3. Installed preflight. talos doctor --start runs eight probes on the installed build and projects the time of a reference turn.
4. Admission. Twelve scenarios, one fresh trial each, captured from a real terminal.
5. Full qualification. 24 scenarios across eight archetypes, from simple tasks to privacy, recovery and context pressure, three repetitions each, 72 fresh trials.

Every trial is checked against eight trust gates, among them approval integrity, mutation scope, completion truth and read-evidence binding. A failed trust gate blocks the candidate before any task score is read. The runbook keeps Talos's own faults apart. "A Talos defect is not a model failure. Record it separately, add a failing regression test, fix it, and start a fresh campaign identity."

The scorecard maps to two tiers. Accepted beta needs at least 20 of 24 scenarios, every archetype at 2 of 3, required-tool accuracy of 0.90, abstention of 0.95, requirement coverage of 0.85 and no timeouts. Experimental selectable needs 16 of 24 scenarios, required-tool accuracy of 0.75 and abstention of 0.90, with looser limits. The scorecard is an input to a human decision, and the product status of a model is set in one place, the model catalog in code. The lifecycle document puts it plainly. "Passing admission is not a support claim."

Where the twelve stopped

All six complete runs scored in the experimental tier. The two models the catalog lists as accepted beta hold that status through an earlier evidence record, made before this protocol.

Most early stops were about the request format

Counting Qwen3.5 9B's first configuration with the five candidates above, four of the six early stops ended on a request the chat template rejected before any token was generated. For the three Ministral models the server returned Jinja Exception: Only user, assistant and tool roles are supported, got system. Talos normally sends its instructions as two system messages, and their templates rejected that shape. For Qwen3.5 9B it returned "System message must be at the beginning." For Qwen3.5 an explicit layout was added that merges only Talos's own system frames into one leading message, and with it Qwen3.5 passed admission 12 of 12. The Ministral models were not retried under the adapter, so their result describes their templates and Talos's request shape, not what the models can do with tools.

The other two models answered, but not with a native tool call. In a one-message probe with server-default sampling and a 256-token cap, Qwen2.5 Coder 7B described a JSON call in prose, added an XML example and hit the cap. The probe did not use the near-greedy settings Talos applies to tool turns, so the rejection covers that configuration only. Its preflight report draws the lesson. "/props proves that the template can represent tools. It does not prove that the model reliably uses the representation." Devstral's template did not advertise tool support at all, and the model wrote read_local_file_tool(input: {"file_path": "README.md"}) as plain text.

What did not fail

These results cover the 432 trials of the six complete runs.

• No trust gate failed.
• No trial timed out, and no final answer reached the output limit. In two Qwen3 4B trials the first 512-token cap cut a tool turn short, and the uncapped retry then looped.
• 745 of 745 native tool calls were schema-valid, and no malformed call went unrecovered in 917 tool-eligible generations.
• Every abstention was correct, 90 of 90.
• 306 of 324 required-tool decisions were correct.

For the six models that went on to a full run, the shape of a tool call stopped being the problem. That is a statement about Talos plus the model, not the model alone. Earlier, stopped attempts are not part of these numbers. In those packets 19 trials failed a trust gate, each recorded at the time as a Talos runtime defect, and at least two were later traced to the validator rather than the runtime.

What did fail

Some failures were in the work.

• Narrow searches. Asked to locate a service port, Granite 4.1 3B in all three trials and Qwen3 4B in one never read config.json. Where a model answered "not found" without reading the right file, Talos replaced the unsupported negative with a bounded disclosure, and the trial was scored as a model failure with every trust gate intact.
• Exact output. Asked to "return only the current status value", Qwen3 8B wrapped the value draft in a full sentence in all three repetitions. The fact was right and the format was not.
• Repetition loops. Qwen3 4B looped on the simplest read scenario in all three repetitions, and the repetition guard aborted each one with an empty answer.
• Unsupported detail. Asked for a cross-document summary, Granite 4.1 3B searched one or two of the three documents with grep, read none of them, and added architectural detail they did not contain. No mechanical check scores added detail, so the scenario failed it for leaving out two required facts and for searching where a file read was required. The read-evidence gate stayed true, because that gate "proves which files were read, not whether every natural-language sentence is entailed by those files."

The six also differed in ways one score hides. Qwen3 4B covered the most scenarios, 20 of 24, and chose the required tool least often, 47 of 54. Granite 4.1 3B was the fastest and lightest, with a median turn of 2.41 seconds and a working set of about 3.2 GiB.

When the harness is part of the result

A later reading of the raw packets found four scenarios whose results are not clean model measurements, two for all six models and two for one or two models each. Those results hold 43 of the 102 failed trials. That does not make all 43 harness failures, since some include model-side misses as well. It means those trials cannot be read as measurements of the model alone. The historical scorecards are unchanged, and no corrected model score follows from these observations.

• The two-part edit, all six models. The scenario asks for a page's heading to become "Atlas Control" and its status paragraph to start with "Offline". Every trial had exactly two model generations, a read and an edit. Both parts were in one file, and Talos ends the tool loop once every expected target has been changed, so after the first successful edit no model was asked for anything more. Five models had sent only the heading edit. Qwen3.5 9B sent both edits in one response, and the scenario's approve-once policy denied the second. Whether the other five would have completed the second part was never observed.
• The static-page explanation, all six models. 15 of the 18 final answers are byte-identical to a template Talos itself renders. The other three are a Talos evidence notice, issued because Granite 4.1 3B did not read the script file. The scenario requires the word "Ready", which the template never contains. Granite's three trials therefore mix a model-side miss with a runtime-rendered answer.
• A two-item privacy answer, Qwen3.5 9B and GPT-OSS 20B. Qwen3.5 numbered its items with two spaces after the number, and the scorer's pattern accepts exactly one. GPT-OSS 20B's one miss on the row is a two-rule answer whose indented sub-bullets the same pattern counted as six items. Qwen3.5's only miss against the accepted-beta thresholds was the scenario count, 19 of 24. Counted, this row would have raised it to 20 of 24 with every other threshold met, and the tier would still be a reviewed decision.
• A yes-or-no question, Granite 4.1 3B. The prompt asks for a yes or no answer, and the scorer compares case-sensitively with No. Granite answered no three times and scored 0 of 3. Nothing in the prompt asked for a capital letter. The evaluator required capitalization that the prompt did not specify, so these three failures do not establish an incorrect yes-or-no answer.

The harness was also tested first. For Qwen2.5 Coder 14B and Qwen3 8B, the halted campaigns stopped because of Talos or its evidence capture, not the model, among them a prompt that asked for an answer "without opening any workspace file" and still produced read calls and a Ctrl-C turned into a second model request. Five of the nine incomplete Granite 4.1 3B full runs also ended on a trial recorded as a Talos runtime defect. Each Talos defect was fixed on the development branch and the campaign restarted under a new identity. The fixes are not yet released, and whether released 0.10.8 has the same defects was not tested, since every campaign ran a later development commit.

Setup

Every packet ran on one desktop, with an NVIDIA RTX 5070 Ti with 16 GB of VRAM, an AMD Ryzen 9 9900X, 128 GB of RAM, Windows 11 and llama.cpp b9918 on CUDA 13.3, with one quantization per model. The usual context was 8,192 tokens. GPT-OSS 20B and the Qwen2.5 Coder 14B admission ran at 16,384, and GPT-OSS 20B's full run placed part of its expert weights on the CPU under a 5,120 MiB allocation limit, which its calibration guide calls "capacity simulation on a 16 GiB host." An older measurement on the same machine shows what the GPU does. A Qwen2.5 Coder 14B code-writing turn took 29.4 seconds on CUDA and hit the 300-second budget on the CPU build.

Turns that must call a tool run near-greedy, with temperature 0.2, top_p 0.8 and top_k 20, after earlier byte-identical requests produced both valid calls and protocol debris under default sampling. They start with an output cap of 512 tokens for reads and 1,024 for changes, and a call cut short is retried once without the cap. The Qwen3 models run with reasoning off. All runs used one backend. Released 0.10.8 already defaults to llama.cpp, and the development branch has since removed Ollama entirely.

No profile carries a Low, Medium or High hardware label. The five profiles this protocol qualified each disclaim a weak-hardware, low-resource or minimum-hardware claim, and the two accepted profiles state RAM minimums no physical hardware packet has measured.

What the evidence supports

On this machine, with this build and these protocols, open-weight models from 3B to 20B passed every trust gate together with Talos and did the work unevenly. Two searched too narrowly, four padded exact answers and one added detail no file supported. GPT-OSS 20B reached the tool-call limit in ten read trials, and outside the two-part edit, twelve change requests across three models ended partial or failed. A large share of the failed trials sat in scenarios shaped by Talos itself, a stop rule, a rendered template and two answer keys, and those trials cannot be read as measurements of the models alone.

• Each scenario ran three times, so the confidence intervals are broad.
• The campaigns ran on different Talos commits, GPT-OSS under protocol v4 and the others under v5. Their own reports call the cross-model tables descriptive, not a controlled benchmark.
• The protocol in the current code is v6, and no model has run it. No qualification has run since these campaigns, while Talos kept changing.
• The two accepted profiles rest on the earlier record. Qwen2.5 Coder 14B has no 72-trial run, and GPT-OSS 20B's request layout has changed since that record.

What transfers

• Probe with the exact request the agent sends, including its system messages and sampling. A template that passed a one-message probe rejected the real request on its first trial.
• Run turns that must call a tool near-greedy with an output cap, and check what the uncapped retry does. Here it fired twice, both for Qwen3 4B, and both retries looped until the repetition guard stopped them.
• Do not let the loop close a multi-part task on the first successful write. Here Talos ended every two-part edit after the first change, and the scorer counted the missing part against the model.
• Separate harness effects from model behavior before scoring. Here 43 of the 102 failed trials were results the runtime or the scorer shaped.
• Score only what the prompt asked for. A key that requires an unstated capital letter measures the key.