Lifelong Language Knowledge Distillation

Yung-Sung Chuang, Shang‐Yu Su, Yun-Nung Chen

Type: Article

Publication Date: 2020-01-01

Citations: 28

DOI: https://doi.org/10.18653/v1/2020.emnlp-main.233

Locations

arXiv (Cornell University) - View - PDF

Similar Works

Action	Title	Year	Authors
+	Lifelong Language Knowledge Distillation	2020	Yung-Sung Chuang Shang‐Yu Su Yun-Nung Chen
+ PDF Chat	GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation	2024	Mohsen Gholami Mohammad Akbari Cindy Hu Vaden Masrani Z. Jane Wang Yong Zhang
+	Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings	2019	Luke Melas-Kyriazi George Han Celine Liang
+	Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings	2020	Luke Melas-Kyriazi George Han Celine Liang
+	LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning	2020	Kaitao Song Hao Sun Xu Tan Tao Qin Jianfeng Lu Hongzhi Liu Tie‐Yan Liu
+	Lifelong Language Pretraining with Distribution-Specialized Experts	2023	Wuyang Chen Yanqi Zhou Nan Du Yanping Huang James Laudon Zhifeng Chen C. C. Iuras ̧cu
+ PDF Chat	L3 Ensembles: Lifelong Learning Approach for Ensemble of Foundational Language Models✱	2024	Aidin Shiri Kaushik Roy Amit Sheth Manas Gaur
+	ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalization	2023	Weixin Liu Xuyi Chen Jiaxiang Liu Shikun Feng Yu Sun Hao Tian Hua Wu
+	L3 Ensembles: Lifelong Learning Approach for Ensemble of Foundational Language Models	2023	Aidin Shiri Kaushik Roy Amit Sheth Manas Gaur
+	Knowledge Inheritance for Pre-trained Language Models	2021	Yujia Qin Yankai Lin Jing Yi Jiajie Zhang Xu Han Zhengyan Zhang Yusheng Su Zhiyuan Liu Peng Li Maosong Sun
+	LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5	2021	Chengwei Qin Shafiq Joty
+	Data-Free Distillation of Language Model by Text-to-Text Transfer	2023	Zheyuan Bai Xinduo Liu Hailin Hu Tianyu Guo Qinghua Zhang Yunhe Wang
+ PDF Chat	Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models	2024	Yao Fu Yu Yin Xuewang Han Ruiteng Li Xianxuan Long Haoran Yu Pan Li
+	MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models	2019	Linqing Liu Huan Wang Jimmy Lin Richard Socher Caiming Xiong
+	Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes	2023	Cheng-Yu Hsieh Chun‐Liang Li Chih‐Kuan Yeh Hootan Nakhost Yasuhisa Fujii Alexander Ratner Ranjay Krishna Chen‐Yu Lee Tomas Pfister
+ PDF Chat	Preserving Knowledge in Large Language Model: A Model-Agnostic Self-Decompression Approach	2024	Zilun Zhang Yutao Sun Tiancheng Zhao Leigang Sha R. Xu Kyusong Lee Jianwei Yin
+ PDF Chat	Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale	2024	Flavio Di Palo Pratibha Singhi Bilal Fadlallah
+	ELLE: Efficient Lifelong Pre-training for Emerging Data	2022	Yujia Qin Jiajie Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou
+ PDF Chat	ELLE: Efficient Lifelong Pre-training for Emerging Data	2022	Yujia Qin Jiajie Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou
+ PDF Chat	ZeroGen: Efficient Zero-shot Learning via Dataset Generation	2022	Jiacheng Ye Jiahui Gao Qintong Li Hang Xu Jiangtao Feng Zhiyong Wu Changyuan Yu Lingpeng Kong

Works That Cite This (18)

Action	Title	Year	Authors
+	Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora	2022	Xisen Jin Dejiao Zhang Henghui Zhu Wei Xiao Shang-Wen Li Xiaokai Wei Andrew Arnold Xiang Ren
+ PDF Chat	Continual Sequence Generation with Adaptive Compositional Modules	2022	Yanzhe Zhang Xuezhi Wang Diyi Yang
+ PDF Chat	Taming the beast : learning to control neural conversational models	2021	Andrea Madotto
+ PDF Chat	Reminding the incremental language model via data-free self-distillation	2022	Han Wang Ruiliu Fu Chengzhang Li Xuejun Zhang Jun Zhou Xing Bai Yonghong Yan Qingwei Zhao
+	Schema-adaptable Knowledge Graph Construction	2023	Hongbin Ye Honghao Gui Xin Xu Xi Chen Huajun Chen Ningyu Zhang
+	Semi-Supervised Lifelong Language Learning	2022	Yingxiu Zhao Yinhe Zheng Bowen Yu Zhiliang Tian Dongkyu Lee Jian Sun Yongbin Li Nevin L. Zhang
+	Continual Knowledge Distillation for Neural Machine Translation	2023	Yuanchi Zhang Peng Li Maosong Sun Yang Liu
+	Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates	2021	Yuqing Xie Yi-An Lai Yuanjun Xiong Yi Zhang Stefano Soatto
+ PDF Chat	Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extraction	2023	Min‐Qian Liu Lifu Huang
+	Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion	2023	Yi-An Lai Elman Mansimov Yuqing Xie Yi Zhang

Works Cited by This (28)

Action	Title	Year	Authors
+	Distilling the Knowledge in a Neural Network	2015	Geoffrey E. Hinton Oriol Vinyals Jay B. Dean
+ PDF Chat	Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems	2015	Tsung-Hsien Wen Milica Gašić Nikola Mrkšić Pei-Hao Su David Vandyke Steve Young
+	Learning without Forgetting	2017	Zhizhong Li Derek Hoiem
+ PDF Chat	Get To The Point: Summarization with Pointer-Generator Networks	2017	Abigail See Peter J. Liu Christopher D. Manning
+	Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning	2017	Victor W. Zhong Caiming Xiong Richard Socher
+	Non-Autoregressive Neural Machine Translation	2017	Jiatao Gu James Bradbury Caiming Xiong Victor O. K. Li Richard Socher
+	Toward Continual Learning for Conversational Agents	2017	Sung‐Jin Lee
+	Born Again Neural Networks	2018	Tommaso Furlanello Zachary C. Lipton Michael Tschannen Laurent Itti Anima Anandkumar
+	Progress & Compress: A scalable framework for continual learning	2018	Jonathan Schwarz Jelena Luketina Wojciech Marian Czarnecki Agnieszka Grabska‐Barwińska Yee Whye Teh Razvan Pascanu Raia Hadsell
+	Lifelong Domain Word Embedding via Meta-Learning	2018	Hu Xu Bing Liu Lei Shu Philip S. Yu