Architecture¶
How PocketLLM is put together, and how it compares to the serving stacks it is usually measured against. Two things are worth knowing before reading:
- The three design documents — Backend unification design, PocketLLM refactor analysis and the cpp_engine multi-backend plan — are three drafts of the same proposal at different points in time. Where they disagree, the newest one wins; none of them describes work that is fully complete.
- cpp_engine vs vLLM/SGLang is a pre-Phase-1 baseline. For the current comparison read PocketLLM vs vLLM vs SGLang.
| Document | What it covers |
|---|---|
| Backend unification design | The original proposal for one engine over swappable device backends. |
| cpp_engine multi-backend refactor plan | The refactor plan that followed it: core/ / engine/ / backends/ layering without giving up per-hardware kernels. |
| PocketLLM refactor analysis (2026-09) | A vLLM/SGLang comparison that motivates the dual-backend design, against master e59d5d3. (Chinese) |
| Feature roadmap for old hardware | What is worth building for 2080 Ti (SM75) and Ascend 910A, and what is not. (Chinese) |
| PocketLLM vs vLLM vs SGLang architecture analysis | The current comparison: scheduling, paged KV, batching, and serving surface. (Chinese) |
| cpp_engine vs vLLM/SGLang comparison | The earlier comparison, retained as the pre-Phase-1 baseline. |
| Ascend 910A performance roadmap | Where the prefill and decode targets actually stand after measurement, and the ranked next steps — including why the decode target needs quantization rather than tuning. |
The invariants these designs exist to protect are stated in the repository's
CLAUDE.md: kernels stay behind the C ABI, and backend selection happens at
configure time rather than in shared code.