Server Memory bandwidth analysis and optimization in Sub-NUMA Clustering mode

High performance computing (HPC) server systems configured in Sub-NUMA Clustering 3-way (SNC3) mode achieve improved memory access efficiency by dividing the CPU into multiple NUMA domains. Intel has integrated diverse on-chip accelerators into its recent flagship datacenter CPU products [1]. A specialized hardware unit called Data Streaming Accelerator (DSA) for accelerating data movement operations and freeing up CPU cycles [2]. During validation, it was observed that the simultaneous READ/WRITE (1R1W) memory traffic to the same cluster (S0,0) via DSA0 consistently produces lower bandwidth, falling significantly short of the expected targets. This issue indicates a performance bottleneck specific to combined 1R1W traffic within the same cluster, directly impacting the achievable memory throughput under such workloads and potentially reducing overall system efficiency in real-world scenarios where READ/WRITE operations target the same memory domain. Several experiments were carefully crafted to identify the root cause and mitigate the risk of performance degradation in customer deployments. Shared in the paper is the process followed, observations and results to improve the bandwidth and meet projections.