TiMePReSt: Time and Memory Efficient Pipeline Parallel DNN Training with
Removed Staleness
TiMePReSt: Time and Memory Efficient Pipeline Parallel DNN Training with
Removed Staleness
DNN training is time-consuming and requires efficient multi-accelerator parallelization, where a single training iteration is split over available accelerators. Current approaches often parallelize training using intra-batch parallelization. Combining inter-batch and intra-batch pipeline parallelism is common to further improve training throughput. In this article, we develop a system, called TiMePReSt, that …