Adaptive sampling‐driven workload balancing for distributed data stream processing

Abstract Partitioning scheme in distributed stream processing systems is a critical factor in processing large‐scale real‐time data streams. Existing solutions either suffer from poor partition quality or high overhead in partitioning time, resulting in severe workload imbalance and backpressure problems. This paper proposes an adaptive sampling‐driven workload balancing (ASWB) scheme to solve the above problem. ASWB introduces two core innovations: (1) an adaptive sampling rate algorithm driven by backpressure signals, which dynamically controls the stream sampling rate to avoid prediction bottlenecks and alleviate workload imbalance, and (2) a high‐frequency key prediction module that leverages a variable‐size window to reduce hash collisions and improve frequency estimation accuracy. This design accelerates the identification of high‐frequency keys within the partition operator, thereby enhancing partitioning quality. We implement ASWB on Apache Flink and evaluate it using large‐scale real‐world datasets. Experimental results show that ASWB improves system throughput by up to 55.03% and reduces processing latency compared with state‐of‐the‐art approaches.