FluidGPU: Fine-Grained Kernel Disaggregation for Large Model Inference on Heterogeneous GPUs

Disaggregation provides a way to utilize heterogeneous GPU clusters, but existing solutions operate at a coarse granularity and are tightly coupled to specific model architectures. FluidGPU is the first kernel disaggregation system designed to improve the performance and cost efficiency of large-model inference on heterogeneous GPUs. It combines offline analysis with online adaptation, extracts precise inter-kernel dependencies from PTX, overlaps communication with computation through pipelined execution, and applies workload-aware scheduling with lightweight runtime adaptation. Across five heterogeneous GPUs and four model architectures, scaling to 16 GPUs, FluidGPU achieves up to 2.3x the serving throughput and 1.6x the cost efficiency of existing disaggregation methods.