Indispensable CPU-centric Checkpointing for GPUs

Checkpointing is a special task in the modern deep learning training process as it poses a hard tradeoff between training efficiency and reliability. Frequent checkpoints of model states can enhance resilience to random system failures, yet obtaining checkpoints must halt the training process to prevent it from updating the model, which, however, inevitably hurts the overall training efficiency. To mitigate this dilemma, the GPUDirect storage (GDS) technique is gaining attention as it enables direct PCIe peer-to-peer accesses between SSDs and GPUs. Conceptually, this technique has the potential of fully exploiting the high PCIe bandwidth to accelerate the checkpointing process. Nevertheless, despite its promising features, GDS has not seen wide deployment in production.