PocketLLM model support

PocketLLM uses model-specific runtimes rather than treating every checkpoint as the same Transformer. The table below describes the current repository state.

One checkpoint has a runtime in this build. The multi-card runtimes left with the cut, and so did the single-card ones that were driven by the C++ front end. Their pages remain as measurement records and are marked Stale in the table; a stale row is a claim about a checkpoint, not an instruction for running it here.

Status definitions

A status is a claim about evidence, so each one names the evidence it stands on rather than asserting a level. The record behind a row is that model's page, and where a run needed its own document it is under performance.

Status	What it means, and what proves it
Text + server	The complete model has generated tokens from a real checkpoint on the stated hardware, and pocketllm serve answers OpenAI chat and completions against it. Proven by the run record on the model page — for V4.1 and MiMo also behind the server, for Qwen by the concurrency validation.
Text, CLI only	Full-model text-in/text-out generation is covered through a CLI or benchmark entrypoint. There is no OpenAI adapter for it.
Experimental	Functionality exists with an explicit caveat on performance, determinism or output parity. The caveat is the ## Known limitations entry that carries it.
Inspect only	Metadata and tensor validation exist without a complete generation runtime: the audit is the evidence, not generated text.
Stale (record only)	The runtime this row's numbers came from is no longer in this repository — it left with the multi-card cut, or it was driven by the retired C++ front end. The page is kept for the measurement, and it says at the top that nothing on it runs here.

Support matrix

The headline result for each model is on its page, together with the conditions it was taken under. This table is the runtime status and nothing else, so a row stays scannable.

Model	Architecture	Format	Runtime	Status	Design
DeepSeek-V4-Flash	MLA + sparse attention + MoE	Safetensors FP4/FP8, GGUF Q2	PyTorch and C++/CUDA, TP4	Text + server	Design
MiniMax-M2.7	GQA + 256-expert MoE	GGUF UD-IQ1_M	Raw-block CUDA, TP4	Text, CLI only	Design
GLM-5.2	DSA/MLA-indexed + dense prefix + MoE	GGUF UD-Q2_K_XL	Raw-block CUDA, TP4	Text, CLI only	Design
Qwen3.8-27B-FP8	48 Gated DeltaNet + 16 GQA	Safetensors FP8 E4M3	C++/CUDA, TP4	Text + server	Design
Qwen3.8-27B-NVFP4	Same text architecture	Safetensors NVFP4 + FP8	C++/CUDA, TP2	Text, CLI only	Design
Qwen3.8-27B (official BF16)	Same text architecture	Safetensors BF16, vision tower	C++/CUDA, audit only; C++/Ascend TP4	Text + server on Ascend TP4	Design
Ternary-Bonsai-2-27B	Same text architecture	GGUF PTQ1_0, 1.75 bits a weight	Stale — was C++/CUDA, one card, no flag	Stale (record only)	Design
DeepSeek-V4.1-Flash	Encoder-decoder, CSA2 shared-KV, MoE	Safetensors FP8 + FP4	--backend v41, host PyTorch, TP4	Text + server	Design
MiMo-V2.6-Flash	9 global + 39 sliding-window, MoE	Safetensors FP8 + MXFP4	--backend mimo, host expert bank, TP4	Text + server	Design
Xing4.0-29B-A4B	MLA + matrix hyper-connection, 64-expert MoE	GGUF IQ4_NL	--backend xing4, one card, experts resident	Text + server	Design

The Design doc column points at the engineering record for that model: what its runtime does and why, the measurements behind each design choice, and the probes those numbers come from. It lives under docs/architecture/, not next to the guides, because it is written for changing the runtime rather than for running it.

Shared baseline

The headline results use 4×RTX 2080 Ti 22 GiB unless the model page says otherwise. Model TPS numbers are not directly comparable unless their checkpoint, prompt, runtime, warm state, and measurement convention match. See Benchmarking.

Adding or updating a model page

Every model has two documents, and they have different readers.

docs/models/<model>.md is the guide. Someone who wants to run the checkpoint reads it, so it holds only what they need, in this order:

1. What the model is, and four spec bullets — backend, parallelism, context, validated hardware.
2. Overview — the architecture table, and the one or two ideas that explain how PocketLLM runs it.
3. Run it — the serve command and a curl, the serving options table, and the non-server entrypoints.
4. What is supported — one table, including the rows that say "not implemented".
5. Performance — one summary table, with the conditions stated above it.
6. Hardware and memory, then Known limitations.
7. Where the detail is — links onward, ending with this matrix.

docs/architecture/<model>_design.md is the record. It holds the kernels, the intermediate measurements, the rejected alternatives, the correctness evidence, the reproduction commands and the source-file inventory. It opens by saying which of its numbers are single readings and which are differences, and it points back at the guide.

Do not put the record on the guide page, and do not claim a capability the record does not establish: never infer runtime support from model metadata alone, because a checkpoint may advertise a long context, a vision tower or an MTP layer that PocketLLM does not execute.