DWI: Efficient On-Device LLM Decoding Through Dynamic Width Selection on Nested Models

Abstract: Deploying large language models on resource-constrained mobile devices remains challenging because of memory limitations and high I/O overhead. DWI combines end-to-end dynamic width selection on nested LLMs, a cyclic-locality-aware cache policy for evicting stale parameters, and a block-wise load-computation pipeline that overlaps in-memory computation with loading from flash. Experiments show that DWI accelerates on-device LLM inference by up to 34x while maintaining competitive model performance.

@article{tmcdwi,
  title={DWI: Efficient On-Device LLM Decoding Through Dynamic Width Selection on Nested Models},
  author={Weining Chen and Yongheng Deng and Xiang Wang and Ziyan Fu and Xiangyu Li and Ting Cao and Yunxin Liu},
  journal={IEEE Transactions on Mobile Computing (TMC)},
  year={2026}
}