Skip to content

Ternary-Bonsai-2-27B

Stale: this repository no longer has a runtime for this checkpoint. Ternary-Bonsai routed through the cpp backend, which was deleted with the multi-card cut — the C++ engine moved to the relic-engine archive, and the Python that drove it is not in this tree. Nothing below runs here today; the command is kept as the record of what was measured and how. What the page is still good for is the checkpoint itself: the 1.75-bit format, the 5.53 GiB footprint, and the 245,760-token context that footprint buys. Re-porting it as a relic-core consumer is on the roadmap.

A 27B hybrid-attention text model — 48 Gated DeltaNet layers and 16 full-attention layers over a dense 17,408-wide MLP — released as a GGUF whose weights are 1.75 bits each. PocketLLM ran the whole checkpoint on one 2080 Ti, behind the same OpenAI-compatible server as its other models.

  • Backend: pocketllm serve, cpp adapter (the default for this checkpoint — no flag needed)
  • Parallelism: 1 GPU
  • Context: the checkpoint declares 262,144; 245,760 tokens is what fits on a 22 GiB card at an FP16 KV cache, and the full 262,144 fits with --kv-cache-dtype fp8
  • Validated on: 1×RTX 2080 Ti 22 GiB, PCIe Gen3

Overview

The checkpoint is prism-ml/Ternary-Bonsai-2-27B-gguf, and its header says what it is: 64 blocks with full_attention_interval = 4, hidden 5120, 24 query heads over 4 KV heads at head length 256, a dense MLP of 17408, and general.architecture = qwen35. That last field is why this model needs no new runtime: PocketLLM's Qwen3.8-27B path already dispatches on it, because the two are the same architecture. The attention, the recurrent layers, the loader, the server, the batching and the prefix cache are the ones already validated for that model.

What is new is the container, and there are two ideas in it.

The weights are ternary, and they are consumed as ternary. Every large matrix in the file is GGML type 143 (PTQ1_0): 128 weights in 28 bytes, one scale per block, 1.75 bits a weight. That is 5.53 GiB for a 27B-parameter model, and there are no high-precision escape hatches behind the label — token_embd and output.weight are ternary too. The kernels expand each block's trits to signed bytes and take an ordinary integer dot product, so nothing falls back to fp16 ten times the size.

The file's weights are pre-rotated, and the activations are rotated to match. Every folded matrix in the file stores W · R⁻¹ for a normalized Sylvester–Walsh–Hadamard transform R, and token_embd is the one tensor that stores R applied rather than inverted. The runtime applies R to the activation before each of those matmuls; the product is what the model computed before rotation, and the rotation is what makes a 1.75-bit weight able to represent that function at all. The transform is declared in the file (prism.hadamard.*), not inferred from a name, and the engine refuses a container whose declaration it cannot honour rather than guessing.

The practical consequence of the packing is the memory: 5.53 GiB of weights leaves a 22 GiB card roughly 15 GiB for KV, which is what a 245,760-token context costs here. The FP8 sibling of this model needs four cards.

One token's path

A decode step reads the whole 5.53 GiB once through a __dp4a GEMV, then runs the 16 full-attention layers over a KV cache that grows with context and the 48 Gated DeltaNet layers over a recurrent state that does not. That split is why decode is nearly flat in context — see the table below — and why the model fits a long context on one card at all: only a quarter of the layers pay for depth.

Run it

Serve it

python -m pocketllm serve \
  --model /path/to/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
  --served-model-name bonsai \
  --max-model-len 8192 \
  --port 8000

--backend is not needed: the adapter registry reads the file's own general.architecture (qwen35), canonicalizes it to qwen3_5, and selects the native engine that claims that name. The checkpoint is one .gguf file and the tokenizer, the special-token ids and the chat template all come out of its header, so nothing else has to be passed and no companion directory is required.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "bonsai", "messages": [{"role": "user", "content": "Hello!"}], "stream": true}'

The endpoint serves /health, /ready, /v1/models, /v1/chat/completions, /v1/completions, SSE streaming, DELETE /v1/requests/{id} and /metrics.

The model must be named as the file. The released directory holds a second artifact beside it (Ternary-Bonsai-2-27B-PQ2_0.gguf), and a directory naming two models is refused rather than resolved by listing order:

UnsupportedFeatureError: the native C++ adapter serves the Qwen3.5 GGUF export, as a single .gguf
file declaring general.architecture=qwen35; other GGUF checkpoints must use backend='torch'

Serving options

Option Default What it does
--max-model-len 8192 Positions the caches hold. The KV arena is sized from this at start, so it is a memory decision as much as a context one: 64 KiB a token.
--kv-cache-dtype auto → fp16 fp8 halves the KV and is what the checkpoint's own 262,144 fits with. It changes the arithmetic, so it is not free.
--enable-prefix-caching on Resumes a prompt that repeats the one just served instead of forwarding it again. The serialized session's only — the batch scheduler's prefill path does not consult it.
--prefill-chunk-tokens 8192 Tokens one prefill call takes at once.
--enable-batching / --max-batch-size on / 8 The batch scheduler. A width above 1 asks for it on its own; --no-enable-batching selects the serialized session and cannot be combined with a width above 1. On by default, at width 8: the width buys aggregate throughput and costs per-request latency — see Known limitations.
--backend-option kv_paged=true off Paged KV blocks instead of one contiguous arena. Memory-neutral on its own, and measured to give up the prefix resume — see Known limitations.
--device-ids first card Cards this process may run on. There is no --tensor-parallel-size in this build — the C++ engine's TP4 is how the FP8 sibling was served, and it went to RelicLLM with the cut.

Without a server

# the engine, all 64 layers, greedy, on the released file: prints the generated ids and text
cpp_engine/build-python/tests/test_qwen_ternary_engine

# the weight map against the FP8 sibling, and the transform against the fork's own bytes
cpp_engine/build-python/tests/test_qwen_gguf_weights
cpp_engine/build-python/tests/test_qwen_hadamard_ops

# the container alone: 851 tensors, read from the file's header
python -m pytest tests/test_ptq1_0_layout.py -q

What is supported

Capability State
OpenAI-compatible serving (chat, completions, streaming, cancel, metrics) Supported, validated on one card
The 1.75-bit PTQ1_0 container, both phases Supported; a ternary tensor is never upcast to fp16
The checkpoint's Hadamard transform, read from the file's declaration Supported
245,760-token context (FP16 KV) / 262,144 (fp8 KV) Supported and measured
Cross-request prefix resume Supported, on by default; one prompt deep
Concurrent requests Serialized by default; the batch scheduler is opt-in and buys 1.14–1.78× aggregate at the cost of per-request latency
Sampling (temperature, top_k, top_p) Supported by the engine; the adapter exposes greedy generation only
MTP, DSpark, DFlash2 speculative decoding Not validated on this checkpoint
Ternary-Bonsai-2-27B-PQ2_0.gguf (type 142, 2.13 bits) Declared in the loader, no kernel — it is refused, not upcast
TP > 1 on CUDA Unmeasured for this artifact
TP4 on the Ascend 910B Supported, validated — see On the Ascend 910B
Vision The chat template renders images; nothing reads them

Performance

One RTX 2080 Ti, CUDA_VISIBLE_DEVICES=0, the released PTQ1_0 file, one request at a time, --max-model-len 32768, greedy. The prompts are real repository documents cut to an exact token count, sent through the checkpoint's own chat template, so prompt tokens below is what the engine counted rather than an estimate.

Result
Prefill, 4,096-token prompt 6.44 s — 636.0 tok/s
Prefill, 8,192-token prompt 12.82 s — 639.1 tok/s
Prefill, 4,097-token prompt 18.11 s — 226.3 tok/s
Decode, at a 4,096-token context 25.9 tok/s — 38.6 ms a token
Decode, at an 8,192-token context 25.4 tok/s
Decode, at a 22,389-token context 24.1 tok/s
A prompt that repeats the one just served TTFT 0.023 s, nothing re-forwarded

Read the first and third rows together. They are the same content, one token apart, and the only difference is that 4,096 is a whole number of 64-token tiles while 4,097 is not. Prefill runs at 636–647 tok/s — flat from 2,048 tokens to 22,378 — except that a prompt whose length is not a multiple of 64 pays a penalty of up to 12 seconds once, in its last partial tile, which is what makes the unaligned row look like a slow rate. Give the engine a multiple of 64 and it is at its real speed; there is no alignment knob, and the penalty is measured and reproducible but not yet diagnosed (#406). See the design document.

Decode falls 7% from a 4,096- to a 22,389-token context — the 48 recurrent layers do not grow with depth and the 16 attention layers do.

scripts/bench_pocketllm_serve_phases.py produced these rows from the server's own /metrics deltas rather than from chunk arrival times; the command is in the design document.

Against the upstream reference, same card

The number that says where this runtime stands is the fork's own llama-bench on the same artifact and the same card, from the reference gate:

PocketLLM, 1 card llama.cpp prism, 1 card
Prefill 636.0 tok/s at 4,096 (639.1 at 8,192) 642.5 tok/s at 4,096 (615.2 at 8,192)
Decode 25.9 tok/s 30.7 tok/s

Prefill is level with the reference — 99% of it at 4,096 tokens and 104% at 8,192, compared aligned to aligned, because llama-bench prefills each prompt in one piece and never pays the ragged-tail penalty above. Decode is 84% of it. Decode is the side with the real gap and the side the 1.75-bit packing was supposed to buy, and the earlier stages measured it doing exactly that (the dense GEMM: 2.4× the FP8-width arm at a tenth of the bytes).

Hardware and memory
Cards 1. Nothing here needs NVLink or a second card.
Weights 5.53 GiB — 402 ternary tensors, one file, 5,946,648,928 bytes
Weights + runtime, no request 6,566 MiB on the card; the 895 MiB above the file is the CUDA context, the engine's arena and the materialized F32/BF16 tensors
KV 64 KiB a token — 16 attention layers × 4 KV heads × 256 × 2 (K and V) × 2 bytes — allocated for the whole --max-model-len at start
Idle at --max-model-len 8192 7,078 MiB
After one cold 4K-token prompt 8,952 MiB, and flat from there — the prefill scratch, cached per thread at its high-water mark
The most that fits 245,760 tokens — 15,360 MiB of KV + 6,566 MiB of everything else, of 22,528 MiB
262,144 tokens Needs --kv-cache-dtype fp8; 15,014 MiB on the card

Both halves of that budget are measured, and the KV half is the same number derived two ways: the architecture gives 64 KiB a token, and four fresh processes at --max-model-len 2,048, 8,192, 16,384 and 32,768 measured the card at 6,694, 7,078, 7,590 and 8,614 MiB — exactly 64 KiB a token above a 6,566 MiB intercept.

The fixed footprint is not the whole story, and the rest of it is bounded. A prefill needs device scratch for the chunk it is working on, and that buffer is cached per thread at its high-water mark rather than reallocated per chunk — so the card does not return to 7,078 MiB after the first long prompt. It settles at 8,952 MiB and stays there: eight cold 4,096-token prompts in one server produced 7,078 → 8,952 MiB, flat after the first. That is the workspace, it is one per thread, and it does not grow with how many prompts the server has answered.

On the Ascend 910B

The same checkpoint runs on four first-generation 910B cards (32 GiB HBM each, CANN 9.0.0), through the same engine and the same weight map. The container, the transform and the tokenizer are backend-independent, so what changes is the arithmetic: the Ascend backend is dense FP16 only, and there is no ternary kernel for it.

The weights are expanded

PTQ1_0 is decoded to FP16 while the tensor is still on the host, so the card holds 27B parameters at two bytes each rather than 1.75 bits: 12.53 GiB resident, against the 5.53 GiB the file occupies. On a 910B that is affordable — the card has 32 GiB where the 2080 Ti has 22 — and it is the only option, because there is no kernel that could consume the blocks in place. The decoder is the same one the CUDA path uses; it just runs unconditionally instead of being skipped when the backend declares it reads packed ternary.

Three further consequences follow from FP16-only and are worth stating because each one is a difference from the CUDA path rather than a property of the model:

  • The embedding table is materialized. The CUDA path reads token_embd's ternary blocks directly. With no packed kernel, the table is decoded like everything else.
  • token_embd's inverse transform is folded into the table rather than applied per lookup, for the same reason.
  • The fold is applied to the weight, not the activation, where the two cannot both happen. A column-parallel shard that stops inside a 1024-element block has no way to rotate its own activation, so the loader unfolds the weight once at load and the site becomes an ordinary dense multiply. At TP4 every row-parallel folded matrix — out_proj, o_proj, down_proj — takes this path. The decision is made in one place (qwen_rotation_needs_weight_unfold) and both the weight map and the materializer read it, so the two cannot disagree about which frame a weight is in.

Four cards, not one

TP4 is how this runs, and TP8 is not available: the model has 4 KV heads and 8 does not divide them, so a TP8 shard would be a partial head. The layout is the one the FP8 sibling uses — column-parallel qkv/gate/up/z, row-parallel out_proj/o_proj/down_proj, and the packed [q | k | v] tensor cut in segments rather than as one range.

What it measures

scripts/run_qwen_ascend_tp4.sh on four 910B cards, the released PTQ1_0 file, greedy:

Result
Load, all 64 layers on 4 cards ~23 s — 497 dense FP16 linears, 12.53 GiB resident
Decode, one row 23.6 tok/s — from a 16-token run, so indicative rather than a rate
Decode, 16 rows batched 173.6 tok/s — 92.2 ms a step
Card memory, one row 19.60 GiB of 32

The load is host work, not disk or device transfer. Both checkpoints put the same 12.53 GiB of FP16 weights on each rank — that is what a TP4 shard of 27B parameters is — and the dense sibling loads it in 35.5 s because it reads FP16 off disk and copies it across. Here the engine reads 5.95 GB instead, decodes 402 ternary tensors to FP16 on the host, and then pushes the same 12.53 GiB — and it now does the whole thing in less time than the sibling does with a copy, because the Hadamard unfold stopped being the load. Three arms on the same command line, two runs each and interleaved, put the previously shipped binary at 123.79 / 122.99 s (model_load_seconds, rank 0), the same code with the unfold restricted to the blocks a shard reads at 63.08 / 62.93 s, and that plus the row fan-out at 22.70 / 24.87 s: 5.1× in all, on bit-identical weights, with the design page carrying the arms, the worker sweep and how the bit-exactness is established.

What the ~23 s goes to
PTQ1_0 decode outside the unfold 11.5 s
Host resize of the 12.53 GiB destination 4.4 s
Device upload, checkpoint read and the rest 4.6 s
Hadamard weight unfold, 128 tensors, 16 workers ~2.7 s

The decode fell because a trit is −1, 0 or 1 and a block's scale is one fp16, so a block's 128 weights take one of exactly three values: the narrowing runs three times a block into a three-entry table and each weight costs a lookup. It is now 2.9 ns a weight, 20.3 s in all. The unfold is no longer the largest term either, and the first three rows above are the previous record's numbers rather than new ones: the change does not touch those regions, and the non-unfold load measures the same 21.5 s it did before, so they are left where they were measured. What is left is the decode and the destination resize, which the design page names as the next two things.

The batching gate

The harness checks the batched path against a synchronous single-row reference at three steps, interleaved across sixteen rows, and compares both the tokens and the logits:

QWEN_BATCH_ROWS=16 QWEN_BATCH_VERIFY=3 scripts/run_qwen_ascend_tp4.sh "The capital of France is" 8

verify_mismatches=0 verify_compared=48 batch_repeat_mismatches=0 seed_mismatches=0
worst_logit_abs=0.0076313

Known limitations

  • The Ascend path expands this checkpoint to FP16. 12.53 GiB resident instead of 5.53, and a ~23 s load. It is the only way to run PTQ1_0 there: there is no packed-ternary kernel for the backend. The expansion is now cheaper than the dense sibling's FP16 load (35.5 s), so the trade is paid off; what remains of the load is the PTQ1_0 decode and the destination resize rather than the Hadamard unfold, which is down to ~2.7 s from the 101 s it was.
  • A prompt whose token count is not a multiple of 64 pays a one-off penalty of up to 12 seconds. The engine's last partial tile is processed by something slow: a 4,096-token prompt prefills in 6.44 s and a 4,097-token one in 18.11 s, same content. The cost tracks the unused slots in that tile — about 0.4 s for each — so it is 12.2 s for a 1-token tail and 1.9 s for a 32-token one, and nothing at all when the tile is full. Prefill beyond it is flat at 1.55 ms a token. It is measured and reproducible on every prompt tried and the mechanism is not yet identified; it is tracked as #406. Note that the chat template adds tokens, so the count that matters is the one the server reports, not the length of the text you sent.
  • The default path batches, and the width costs a lone request some latency. With --backend-option enable_batching=true --backend-option max_batch_size=4 the aggregate decode rate rises 1.78× at a short prompt and 1.14× at a 2,044-token one, while per-request latency grows with the batch — at 2,044 tokens the batched group's median is 41.1 s where the serial queue's median is 35.7 s, so four callers wait longer in total than they would have queued. The scheduler runs the width's rows whether or not that many requests are present, which is the same effect seen from the other side: one request alone through slots=2 pays more than it would through the serialized session, and through slots=8 more still. The scheduler also does not consult the prefix cache, so a prompt the serialized session would have resumed is re-forwarded in full — the larger of the two costs for a caller that repeats its prompt. Turn the width down (--max-batch-size 2) or off (--no-enable-batching) for single-caller latency, and up for aggregate throughput. The per-request cost of the width, measured with the prompt cache held fixed, is in the concurrency acceptance page.
  • An earlier build of this branch reported that a greedy answer could depend on the batch. Four identical requests produced two distinct texts, and a prompt choosing between "5:00:00" and "0:15:00" gave one row each. That measurement came from a build whose greedy runs did not stop at </s> on those prompts — all 64 tokens were "!" — and on the current build the same probe answers identically across all four rows, with the prefix cache on and off and over repeated runs. The caveat is therefore not claimed here; the earlier observation is unexplained rather than withdrawn, and it is recorded in the design document.
  • --backend-option kv_paged=true gives up the prefix resume. Measured: with paging on, a repeated prompt was forwarded again in full every time, where the default arena answers the repeat from the stored state. Decode is also about 3% slower (25.0 against 25.8 tok/s). Paging exists for the case where one arena will not do; it is not the better default here.
  • Prefix reuse is one prompt deep. The engine keeps the state of the request it just served, so a repeat of that prompt resumes and a repeat of an earlier one does not. Sending prompt A, then B, then A again forwards A in full the second time. It is a resume, not a store.
  • Decode is 84% of the reference on the same card (25.9 against 30.7 tok/s). This is inherited from the dense-GEMM stage, not introduced by the server, and it is where the format's remaining work is: the packing already buys 2.4× against the FP8-width arm at a tenth of the bytes.
  • Speculative decoding is not validated here. MTP, DSpark and DFlash2 exist in this engine for the FP8 checkpoint; none was run against the ternary artifact, and this record does not claim that the flags are wired to anything for it.
  • The adapter is greedy only. temperature, top_k and top_p are accepted by the native engine but the C++ adapter refuses a non-greedy request rather than silently ignoring it.
  • PQ2_0 cannot be served. The loader decodes its geometry and refuses it by name; there is no kernel for type 142.

Where the detail is