A model you already have, and an architecture you can edit

The local provider runs a quantized model inside the Synsema process: no server, no API key, no socket — the only provider that works with network access denied outright. That existed before. What v0.6.27 changes is the two things that made it awkward to start: you had to find a .gguf and give it a path, and a model whose architecture we had not compiled in was simply out of reach.

The model is a name, and nothing is downloaded§

SYNSEMA_LLM_PROVIDER=local

SYNSEMA_LLM_MODEL=/models/qwen2.5-3b-instruct-q4_k_m.gguf   # a path to a .gguf
SYNSEMA_LLM_MODEL=qwen3:0.6b                                # a model:tag already in the Ollama cache
SYNSEMA_LLM_MODEL=org/repo                                  # a repo already in the Hugging Face cache

None of the three fetches a byte. If you use Ollama you are already done — ollama does not even need to be running, only its files on disk are read. A name that is not there lists the three places it looked, with real paths, instead of starting a download you did not ask for.

synsema llm status says what is actually on the machine:

Modelos locales ya descargados (2):
  gemma3:270m                            ollama:gemma3:270m  sha256:735af2139dc6
  qwen3:0.6b                             ollama:qwen3:0.6b  sha256:7f4030143c1c
  Usá SYNSEMA_LLM_MODEL=<nombre> — no se descarga nada.

The sha256 is free: Ollama's store is content-addressed, so the hash of the weights comes out of the filename rather than out of a second read. It matters later, when you want to say which weights answered.

Architectures are a file, not a release§

Every runtime that runs GGUF — llama.cpp, candle, Ollama — writes each architecture by hand, in its own language. Adding one means code, a pull request, a review and a release. Our own three-line PR to candle, approved, sat open for over two months, and two quantized architectures we needed sat with it. Copying that model would make us the bottleneck for anyone who wants to run a model we have not gotten to.

So in the engine written by us, an architecture is a text file the already-compiled binary reads at startup. The operations — matmul, RMSNorm, RoPE, attention, SwiGLU — are compiled in; what was missing is the order and the parameters, and those are data:

arch qwen3
kind decoder

prologue
  x = embed(token_embd.weight)

block
  h = rms_norm(x, blk.{i}.attn_norm.weight)
  q = matmul(h, blk.{i}.attn_q.weight)
  k = matmul(h, blk.{i}.attn_k.weight)
  v = matmul(h, blk.{i}.attn_v.weight)
  norm_heads(q, blk.{i}.attn_q_norm.weight, head_count)     # <- what qwen3 adds to llama
  norm_heads(k, blk.{i}.attn_k_norm.weight, head_count_kv)
  rope(q, head_count)
  rope(k, head_count_kv)
  a = attention(q, k, v)
  o = matmul(a, blk.{i}.attn_output.weight)
  add(x, o)

block runs once per layer with {i} replaced by the index. Tensors are named exactly as the GGUF names them, so writing a definition is mostly copying the tensor list out of the file, and what the GGUF already declares — head counts, embedding length, the sliding window — is not repeated. Four definitions ship embedded in the binary (llama, qwen2, qwen3, gemma3); SYNSEMA_INFER_ARCHDEF=<dir> adds yours, or replaces ours, with no compiler anywhere:

SYNSEMA_INFER_BACKEND=rust
SYNSEMA_INFER_ARCHDEF=./archdefs

Arquitecturas que corre el backend `rust` (4):
  gemma3 (20 pasos por capa, en el binario, sha 9e02d6e4ca46)
  llama (16 pasos por capa, en el binario, sha da3951d21c0d)
  qwen2 (19 pasos por capa, en el binario, sha d48ca19bf049)
  qwen3 (18 pasos por capa, ./archdefs/qwen3.archdef, sha 5b8d9db76a68)

A file of yours with the name of one of ours wins, and the line shows it with its path and sha, so a substitution is never silent.

No control flow, and that is the security property§

A definition has no conditionals, no loops, no function calls, and no way to open a file, a socket or the environment. It describes a graph of matrix multiplications. Running a definition somebody else wrote does not run their code: the worst a hostile one can do is fail to load, or give wrong numbers with your weights, under the same resource limits as any model. That is what makes it reasonable to accept a definition from a stranger, and it is exactly why native plugins (.so/.dll) were never an option here — those would be remote code execution with extra steps.

A test enforces it: twenty-one words including if, while, for, exec, import, open, http and env are rejected as operations that do not exist. The day control flow is added the property is gone, so it is not getting added.

And a broken file never falls back to ours. The first version did, and with a typo in silu the model answered perfectly — using our definition. Whoever wrote the file would have sworn theirs was running. Now that architecture is unavailable and the error names the file and the line:

[local error: no se pudo cargar 'qwen3:0.6b': el modelo declara la arquitectura 'qwen3', que este
binario no conoce.
Conocidas: gemma3, llama, qwen2.
Definiciones que no cargaron:
  - ./archdefs/qwen3.archdef: línea 28: no existe la operación `siluu` — ¿quisiste decir `silu`?]

Two engines, and why you would switch§

The SIMD row is the one that shows up as a number on an ordinary machine. candle picks its AVX2 kernels with #[cfg(target_feature)], and the default x86-64 target does not enable AVX2, so the binary you download runs the scalar path unless it is rebuilt for your CPU — a rebuild that measured ~3.1× on prefill. The rust engine dispatches on the CPU it finds, so the official binary uses the instructions of the machine it lands on.

The RAM row is what lets a model larger than your memory run at all: with the weights mapped and kept quantized, a 523 MB GGUF costs 583 MB resident against candle's 1,355 MB. It pages, it is slow, it finishes. Generation is still 1.37× slower than candle (11.1 s vs 8.1 s for 40 tokens, measured on the same file) because token by token the matmul is a single row, and what dominates there is moving memory.

Switching engines changes the generated text. Both are correct — they approximate the same numbers differently — so the engine is part of what you declare to reproduce an output, next to the binary and the weights. candle stays the default while both exist.

If you ran a llama, Mistral or Gemma GGUF before v0.6.27, its answers were wrong§

This is the fix worth reading even if none of the above interests you. GGUF files whose tokenizer.ggml.model is llama — llama 1 and 2, Mistral, Gemma — store token ranks, not log-probabilities, and the tokenizer was segmenting them with a Viterbi pass that maximises a sum of scores. The capital of France is entered the model as eleven fragments instead of five words. Nothing failed. The model just answered badly, and there was no way to tell that from the output.

It is now the reference algorithm — merge the neighbouring pair with the best score, as llama.cpp does — with literal recognition of special tokens, byte fallback and add_space_prefix read from the metadata. The BPE family (qwen, llama 3) was never affected.

Gemma 3 had two more: its MLP activation was silu and it is gelu_pytorch_tanh (copied from candle's quantized_gemma3, which hardcodes silu while its own non-quantized gemma3 reads the config), and <start_of_turn> was not recognised, so a gemma GGUF fell back to plain mode and behaved like a base model. With the three fixed, the engine generates token for token what Ollama generates for the same prompt.

What it costs, honestly§

It is built for short prompts. CPU prefill is ~12 tok/s, so a 1000-token prompt takes ~90 s on a 0.5B; generation is ~11 tok/s on a 0.5B and ~5 tok/s on a 3B with four threads. Model load — 7 s for a 0.5B, ~35 s for a 3B — is paid once per process, so under serve the first request loads it and the rest reuse it (measured: 8.2 s → 1.3 s).

Which is to say: for a classifier, a router, a rewriter, a summary of three paragraphs, on a machine with no GPU and no egress, it is the right tool. For a 4,000-token prompt with a long answer on CPU it is not, and no amount of configuration will make it be.

Here is the whole thing on a laptop, with a model that was already in the Ollama cache:

$ SYNSEMA_LLM_PROVIDER=local SYNSEMA_LLM_MODEL=gemma3:270m SYNSEMA_INFER_BACKEND=rust synsema run probe.syn
1) Apple
2) Sweet
tokens: 35
provider vivo: true

5.4 s of wall clock for two reason calls, model load included. Nothing downloaded, no key anywhere.

Saying what ran§

With SYNSEMA_LLM_TEMPERATURE=0 (the default) generation is greedy and repeatable. To state what exactly produced an output, three things have to travel together, and synsema llm status --json carries all three under an inference key, with full shas:

Provenance is only useful if it can be compared, and comparing prose is not comparing.

Who chooses§

The .syn program never names a model, an architecture or a path, and it cannot make the engine read a file the configuration did not enable. It asks to generate text; everything above is deployment configuration. Discovering the Ollama and Hugging Face caches offers candidates to whoever writes that configuration — it does not open the disk to the program.

The manual page is Local inference. The judge slot got the same treatment in the same release — a checkpoint on disk answering typed questions with no network and no key — and that is Run the judge locally with Laya.