Direct Differentiation of Non-blocking MPI Library Calls

Most applications in high-performance computing require distributing tasks across multiple processors or cores. One common approach to achieve this distribution is using the MPI library, which provides the infrastructure for enabling communication between processes. The MPI standard is continuously evolving, and the library enjoys widespread use. Many applications, for example in optimization, equation solving, or sensitivity analysis, require derivatives. These derivatives can be computed using finite differences or through code generated by Algorithmic Differentiation (AD). Source-to-source transformation of AD generates code to compute the derivatives from the given code to compute the function itself. If the function’s code includes MPI library calls, the AD tool must be capable of handling these appropriately. Unlike other parallel computing strategies such as OpenMP, OpenACC, CUDA, or co-array Fortran, MPI is a low-level library that is linked, but its calls are typically not analyzed by the compiler. In this paper, we provide a detailed description of how a program that uses several widely-used non-blocking MPI library calls is transformed into a parallel program capable of computing derivatives.