Automatic Graph Partitioning for Very Large-scale Deep Learning
Automatic Graph Partitioning for Very Large-scale Deep Learning
This work proposes RaNNC (Rapid Neural Network Connector) as middleware for automatic hybrid parallelism. In recent deep learning research, as exemplified by T5 and GPT-3, the size of neural network models continues to grow. Since such models do not fit into the memory of accelerator devices, they need to be …