FluidGPU: Fine-Grained Kernel Disaggregation for Large Model Inference on Heterogeneous GPUs

Abstract: Disaggregation provides a way to utilize heterogeneous GPU clusters, but existing solutions operate at a coarse granularity and are tightly coupled to specific model architectures. FluidGPU is the first kernel disaggregation system designed to improve the performance and cost efficiency of large-model inference on heterogeneous GPUs. It combines offline analysis with online adaptation, extracts precise inter-kernel dependencies from PTX, overlaps communication with computation through pipelined execution, and applies workload-aware scheduling with lightweight runtime adaptation. Across five heterogeneous GPUs and four model architectures, scaling to 16 GPUs, FluidGPU achieves up to 2.3x the serving throughput and 1.6x the cost efficiency of existing disaggregation methods.

@inproceedings{scfluidgpu,
  title={FluidGPU: Fine-Grained Kernel Disaggregation for Large Model Inference on Heterogeneous GPUs},
  author={Tiancheng Hu and Jin Qin and Lei Chen and Junhao Hu and Yuzheng Wang and Mingxing Zhang and Chunwei Xia and Yizhou Shan and Huimin Cui and Ting Cao and Zheng Wang and Tao Xie and Chenxi Wang},
  booktitle={International Conference for High Performance Computing, Networking, Storage, and Analysis (SC)},

  year={2026}
}