Yufeng Lyu

Yufeng Lyu

AI Framework Researcher at BAAI. Local LLMs, Torch-FL, and efficient inference.

Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline

Published in ASPLOS 2025, 2025

Recommended citation: Fang, Zhiyuan, Yuegui Huang, Zicong Hong, Yufeng Lyu, Wuhui Chen, Yue Yu, Fan Yu, and Zibin Zheng. "Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline." In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 574–588. 2025. https://arxiv.org/abs/2502.06888

Klotski is an MoE inference engine that reduces pipeline bubbles with an expert-aware multi-batch pipeline, a constraint-sensitive I/O–compute planner, and a correlation-aware expert prefetcher. Experiments report up to 85.12× throughput improvement over prior offloading baselines.

I co-designed the expert-aware prefetch method and the expert offload mechanism as part of the CAAI–MindSpore academic research program.