PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management

Jiarui Fang, Yu Yang, Zilin Zhu, Shenggui Li, Yang You, Jie Zhou

Type: Preprint

Publication Date: 2021-01-01

Citations: 3

DOI: https://doi.org/10.48550/arxiv.2108.05818

View Publication

Locations

arXiv (Cornell University) - View - PDF
DataCite API - View

Similar Works

Action	Title	Year	Authors
+ PDF Chat	Parallel Training of Pre-Trained Models via Chunk-Based Dynamic Memory Management	2022	Jiarui Fang Zilin Zhu Shenggui Li Hui Su Yang Yu Jie Zhou Yang You
+ PDF Chat	BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Training	2024	Houming Wu Ling Chen Wenjie Yu
+	Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models	2022	Zhiquan Lai Shengwei Li Xudong Tang Keshi Ge Weijie Liu Yabo Duan Linbo Qiao Dongsheng Li
+	Whale: A Unified Distributed Training Framework.	2020	Ang Wang Xianyan Jia Le Jiang Jie Zhang Yong Li Wei Lin
+ PDF Chat	Understanding and optimizing packed neural network training for hyper-parameter tuning	2021	Rui Liu Sanjay Krishnan Aaron J. Elmore Michael J. Franklin
+	Understanding and Optimizing Packed Neural Network Training for Hyper-Parameter Tuning	2020	Rui Liu Sanjay Krishnan Aaron J. Elmore Michael J. Franklin
+ PDF Chat	Improving Automatic Parallel Training via Balanced Memory Workload Optimization	2024	Y X Wang Youhe Jiang Xupeng Miao Fangcheng Fu Shenhan Zhu Xiaonan Nie Yaofeng Tu Bin Cui
+	Improving Automatic Parallel Training via Balanced Memory Workload Optimization	2023	Yujie Wang Youhe Jiang Xupeng Miao Fangcheng Fu Shenhan Zhu Xiaonan Nie Yaofeng Tu Bin Cui
+ PDF Chat	Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator	2024	Kazuki Fujii Kohei Watanabe Rio Yokota
+ PDF Chat	Angel-PTM: A Scalable and Economical Large-Scale Pre-Training System in Tencent	2023	Xiaonan Nie Yi Liu Fangcheng Fu Jinbao Xue Dian Jiao Xupeng Miao Yangyu Tao Bin Cui
+	Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent	2023	Xiaonan Nie Yi Liu Fangcheng Fu Jinbao Xue Dian Jiao Xupeng Miao Yangyu Tao Bin Cui
+	Whale: Scaling Deep Learning Model Training to the Trillions.	2020	Xianyan Jia Le Jiang Ang Wang Jie Zhang Xinyuan Li Wencong Xiao Langshi chen Yong Li Zhen Zheng Xiaoyong Liu
+	Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform	2018	Chi‐Chung Chen Chia-Lin Yang Hsiang-Yun Cheng
+ PDF Chat	Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training	2023	Shenggui Li Hongxin Liu Zhengda Bian Jiarui Fang Haichen Huang Yuliang Liu Boxiang Wang Yang You
+ PDF Chat	GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching	2024	Cong Guo Rui Zhang Jiale Xu Jingwen Leng Zihan Liu Ziyu Huang Minyi Guo Hao Wu Shouren Zhao Junping Zhao
+ PDF Chat	HetHub: A Heterogeneous distributed hybrid training system for large-scale models	2024	Si Xu Zixiao Huang Yan Zeng Shengen Yan Xuefei Ning Haolin Ye Sipei Gu Chunsheng Shui Zhezheng Lin Hao Zhang
+ PDF Chat	Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training	2019	Saptadeep Pal Eiman Ebrahimi Arslan Zulfiqar Yaosheng Fu Victor Zhang Szymon Migacz David Nellans Puneet Gupta
+	GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching	2024	Cong Guo Rui Zhang Jiale Xu Jingwen Leng Zihan Liu Ziyu Huang Minyi Guo Hao Wu Shouren Zhao Zhao Jun-ping
+ PDF Chat	PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training	2024	Daiyaan Arfeen Zhen Zhang Xinwei Fu Gregory R. Ganger Yida Wang
+	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	Li Shen Yanli Zhao Rohan Varma Omkar Salpekar Pieter Noordhuis Teng Li Adam Paszke Jeff Smith Brian Vaughan Pritam Damania

Works That Cite This (1)

Action	Title	Year	Authors
+ PDF Chat	Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training	2023	Shenggui Li Hongxin Liu Zhengda Bian Jiarui Fang Haichen Huang Yuliang Liu Boxiang Wang Yang You

Works Cited by This (0)

Action	Title	Year	Authors