# A model you already have, and an architecture you can edit > Since v0.6.27 the embedded local provider takes the name of a model already in your Ollama or Hugging Face cache — nothing is downloaded — and an architecture is a text file the compiled binary reads, so adding a model no longer waits for a release of ours. Published 2026-09-22 · https://synsema.org/blog/local-inference-architectures-as-files The `local` provider runs a quantized model **inside the Synsema process**: no server, no API key, no socket — the only provider that works with network access denied outright. That existed before. What **v0.6.27** changes is the two things that made it awkward to start: you had to find a `.gguf` and give it a path, and a model whose architecture we had not compiled in was simply out of reach. ## The model is a name, and nothing is downloaded ``` SYNSEMA_LLM_PROVIDER=local SYNSEMA_LLM_MODEL=/models/qwen2.5-3b-instruct-q4_k_m.gguf # a path to a .gguf SYNSEMA_LLM_MODEL=qwen3:0.6b # a model:tag already in the Ollama cache SYNSEMA_LLM_MODEL=org/repo # a repo already in the Hugging Face cache ``` **None of the three fetches a byte.** If you use Ollama you are already done — `ollama` does not even need to be running, only its files on disk are read. A name that is not there lists the three places it looked, with real paths, instead of starting a download you did not ask for. `synsema llm status` says what is actually on the machine: ``` Modelos locales ya descargados (2): gemma3:270m ollama:gemma3:270m sha256:735af2139dc6 qwen3:0.6b ollama:qwen3:0.6b sha256:7f4030143c1c Usá SYNSEMA_LLM_MODEL= — no se descarga nada. ``` The sha256 is free: Ollama's store is content-addressed, so the hash of the weights comes out of the filename rather than out of a second read. It matters later, when you want to say *which* weights answered. ## Architectures are a file, not a release Every runtime that runs GGUF — llama.cpp, candle, Ollama — writes each architecture **by hand, in its own language**. Adding one means code, a pull request, a review and a release. Our own three-line PR to candle, approved, sat open for over two months, and two quantized architectures we needed sat with it. Copying that model would make **us** the bottleneck for anyone who wants to run a model we have not gotten to. So in the engine written by us, an architecture is **a text file the already-compiled binary reads at startup**. The operations — matmul, RMSNorm, RoPE, attention, SwiGLU — are compiled in; what was missing is the *order* and the parameters, and those are data: ``` arch qwen3 kind decoder prologue x = embed(token_embd.weight) block h = rms_norm(x, blk.{i}.attn_norm.weight) q = matmul(h, blk.{i}.attn_q.weight) k = matmul(h, blk.{i}.attn_k.weight) v = matmul(h, blk.{i}.attn_v.weight) norm_heads(q, blk.{i}.attn_q_norm.weight, head_count) # <- what qwen3 adds to llama norm_heads(k, blk.{i}.attn_k_norm.weight, head_count_kv) rope(q, head_count) rope(k, head_count_kv) a = attention(q, k, v) o = matmul(a, blk.{i}.attn_output.weight) add(x, o) ``` `block` runs once per layer with `{i}` replaced by the index. Tensors are named **exactly as the GGUF names them**, so writing a definition is mostly copying the tensor list out of the file, and what the GGUF already declares — head counts, embedding length, the sliding window — is not repeated. Four definitions ship embedded in the binary (`llama`, `qwen2`, `qwen3`, `gemma3`); `SYNSEMA_INFER_ARCHDEF=` adds yours, or **replaces** ours, with no compiler anywhere: ``` SYNSEMA_INFER_BACKEND=rust SYNSEMA_INFER_ARCHDEF=./archdefs ``` ``` Arquitecturas que corre el backend `rust` (4): gemma3 (20 pasos por capa, en el binario, sha 9e02d6e4ca46) llama (16 pasos por capa, en el binario, sha da3951d21c0d) qwen2 (19 pasos por capa, en el binario, sha d48ca19bf049) qwen3 (18 pasos por capa, ./archdefs/qwen3.archdef, sha 5b8d9db76a68) ``` A file of yours with the name of one of ours **wins**, and the line shows it with its path and sha, so a substitution is never silent. ### No control flow, and that is the security property A definition has **no conditionals, no loops, no function calls, and no way to open a file, a socket or the environment**. It describes a graph of matrix multiplications. Running a definition somebody else wrote does **not** run their code: the worst a hostile one can do is fail to load, or give wrong numbers with *your* weights, under the same resource limits as any model. That is what makes it reasonable to accept a definition from a stranger, and it is exactly why native plugins (`.so`/`.dll`) were never an option here — those would be remote code execution with extra steps. A test enforces it: twenty-one words including `if`, `while`, `for`, `exec`, `import`, `open`, `http` and `env` are rejected as operations that do not exist. The day control flow is added the property is gone, so it is not getting added. And a broken file **never falls back to ours**. The first version did, and with a typo in `silu` the model answered perfectly — using our definition. Whoever wrote the file would have sworn theirs was running. Now that architecture is unavailable and the error names the file and the line: ``` [local error: no se pudo cargar 'qwen3:0.6b': el modelo declara la arquitectura 'qwen3', que este binario no conoce. Conocidas: gemma3, llama, qwen2. Definiciones que no cargaron: - ./archdefs/qwen3.archdef: línea 28: no existe la operación `siluu` — ¿quisiste decir `silu`?] ``` ## Two engines, and why you would switch | | `candle` (default) | `SYNSEMA_INFER_BACKEND=rust` | |---|---|---| | Architectures | llama, qwen2, qwen3 — compiled in | the same plus **gemma3**, each one a file you can read and replace | | A new architecture | needs a new binary | needs a **text file** | | SIMD | chosen at **compile time**: a plain build runs the scalar path | chosen at **run time** — AVX, AVX2+FMA, AVX-512, NEON | | RAM | ~2.6× the size of the `.gguf` | **~1.1×** — the file is memory-mapped and the weights stay quantized | The SIMD row is the one that shows up as a number on an ordinary machine. candle picks its AVX2 kernels with `#[cfg(target_feature)]`, and the default x86-64 target does not enable AVX2, so the binary you download runs the scalar path unless it is rebuilt for your CPU — a rebuild that measured **~3.1×** on prefill. The `rust` engine dispatches on the CPU it finds, so the official binary uses the instructions of the machine it lands on. The RAM row is what lets a model larger than your memory **run at all**: with the weights mapped and kept quantized, a 523 MB GGUF costs 583 MB resident against candle's 1,355 MB. It pages, it is slow, it finishes. Generation is still 1.37× slower than candle (11.1 s vs 8.1 s for 40 tokens, measured on the same file) because token by token the matmul is a single row, and what dominates there is moving memory. **Switching engines changes the generated text.** Both are correct — they approximate the same numbers differently — so the engine is part of what you declare to reproduce an output, next to the binary and the weights. candle stays the default while both exist. ## If you ran a llama, Mistral or Gemma GGUF before v0.6.27, its answers were wrong This is the fix worth reading even if none of the above interests you. GGUF files whose `tokenizer.ggml.model` is `llama` — llama 1 and 2, Mistral, Gemma — store token **ranks**, not log-probabilities, and the tokenizer was segmenting them with a Viterbi pass that maximises a sum of scores. `The capital of France is` entered the model as **eleven fragments instead of five words**. Nothing failed. The model just answered badly, and there was no way to tell that from the output. It is now the reference algorithm — merge the neighbouring pair with the best score, as llama.cpp does — with literal recognition of special tokens, byte fallback and `add_space_prefix` read from the metadata. The BPE family (qwen, llama 3) was never affected. Gemma 3 had two more: its MLP activation was `silu` and it is `gelu_pytorch_tanh` (copied from candle's `quantized_gemma3`, which hardcodes `silu` while its own non-quantized `gemma3` reads the config), and `` was not recognised, so a gemma GGUF fell back to plain mode and behaved like a base model. With the three fixed, the engine generates token for token what Ollama generates for the same prompt. ## What it costs, honestly It is built for **short prompts**. CPU prefill is ~12 tok/s, so a 1000-token prompt takes ~90 s on a 0.5B; generation is ~11 tok/s on a 0.5B and ~5 tok/s on a 3B with four threads. Model load — 7 s for a 0.5B, ~35 s for a 3B — is paid **once per process**, so under `serve` the first request loads it and the rest reuse it (measured: 8.2 s → 1.3 s). Which is to say: for a classifier, a router, a rewriter, a summary of three paragraphs, on a machine with no GPU and no egress, it is the right tool. For a 4,000-token prompt with a long answer on CPU it is not, and no amount of configuration will make it be. Here is the whole thing on a laptop, with a model that was already in the Ollama cache: ``` $ SYNSEMA_LLM_PROVIDER=local SYNSEMA_LLM_MODEL=gemma3:270m SYNSEMA_INFER_BACKEND=rust synsema run probe.syn 1) Apple 2) Sweet tokens: 35 provider vivo: true ``` 5.4 s of wall clock for two `reason` calls, model load included. Nothing downloaded, no key anywhere. ## Saying what ran With `SYNSEMA_LLM_TEMPERATURE=0` (the default) generation is greedy and repeatable. To state *what exactly* produced an output, three things have to travel together, and `synsema llm status --json` carries all three under an `inference` key, with full shas: | | Where it comes from | |---|---| | the weights | `models_on_disk[].digest` | | the architecture | `architectures[].sha256` and `.origin` | | the engine | `backend` — candle and `rust` produce different text from the same weights | Provenance is only useful if it can be compared, and comparing prose is not comparing. ## Who chooses The `.syn` program never names a model, an architecture or a path, and it cannot make the engine read a file the configuration did not enable. It asks to generate text; everything above is deployment configuration. Discovering the Ollama and Hugging Face caches offers candidates to whoever writes that configuration — it does not open the disk to the program. The manual page is [Local inference](https://synsema.dev/en/0.6.x/52a-local-inference). The judge slot got the same treatment in the same release — a checkpoint on disk answering typed questions with no network and no key — and that is [Run the judge locally with Laya](/blog/run-the-judge-locally-with-laya).