MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks

Zhiqiang Shen, Marios Savvides

Type: Preprint

Publication Date: 2020-01-01

Citations: 49

DOI: https://doi.org/10.48550/arxiv.2009.08453

View Publication

Locations

arXiv (Cornell University) - View
DataCite API - View

Similar Works

Action	Title	Year	Authors
+ PDF Chat	Knowledge distillation: A good teacher is patient and consistent	2022	Lucas Beyer Xiaohua Zhai Amélie Royer Larisa Markeeva Rohan Anil Alexander Kolesnikov
+	Knowledge distillation: A good teacher is patient and consistent	2021	Lucas Beyer Xiaohua Zhai Amélie Royer Larisa Markeeva Rohan Anil А. И. Колесников
+	VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale	2023	Zhiwei Hao Jianyuan Guo Kai Han Hu Han Chang Xu Yunhe Wang
+	Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones	2021	Cheng Cui Ruoyu Guo Yuning Du Dongliang He Fu Li Zewu Wu Qiwen Liu Shilei Wen Jizhou Huang Xiaoguang Hu
+ PDF Chat	Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones	2021	Cheng Cui Ruoyu Guo Yuning Du Dongliang He Fu Li Zewu Wu Qiwen Liu Shilei Wen Jizhou Huang Xiaoguang Hu
+	Knowledge Distillation by On-the-Fly Native Ensemble	2018	Lan Xu Xiatian Zhu Shaogang Gong
+	Knowledge Distillation by On-the-Fly Native Ensemble	2018	Lan Xu Xiatian Zhu Shaogang Gong
+	Meta Knowledge Distillation	2022	Jihao Liu Boxiao Liu Hongsheng Li Yu Liu
+ PDF Chat	From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels	2023	Zhendong Yang Ailing Zeng Zhe Li Tianke Zhang Chun Yuan Yu Li
+	From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels	2023	Zhendong Yang Ailing Zeng Zhe Li Tianke Zhang Chun Yuan Yu Li
+	DOT: A Distillation-Oriented Trainer	2023	Borui Zhao Quan Cui Renjie Song Jiajun Liang
+ PDF Chat	Accessing Vision Foundation Models at ImageNet-level Costs	2024	Yitian Zhang Xu Ma Yue Bai Huan Wang Yun Fu
+ PDF Chat	SpectralKD: Understanding and Optimizing Vision Transformer Distillation through Spectral Analysis	2024	Huiyuan Tian Bonan Xu Shijian Li Gang Pan
+	Generic-to-Specific Distillation of Masked Autoencoders	2023	Wei Huang Zhiliang Peng Dong Li Furu Wei Jianbin Jiao Qixiang Ye
+	Co-advise: Cross Inductive Bias Distillation	2022	Sucheng Ren Zhengqi Gao Tianyu Hua Zihui Xue Yonglong Tian Shengfeng He Hang Zhao
+ PDF Chat	ScaleKD: Strong Vision Transformers Could Be Excellent Teachers	2024	Jiawei Fan Chao Li Xiaolong Liu Anshou Yao
+ PDF Chat	ReffAKD: Resource-efficient Autoencoder-based Knowledge Distillation	2024	Divyang Doshi Jungeun Kim
+ PDF Chat	Generic-to-Specific Distillation of Masked Autoencoders	2023	Wei Huang Zhiliang Peng Dong Li Furu Wei Jianbin Jiao Qixiang Ye
+ PDF Chat	Masked Autoencoders Enable Efficient Knowledge Distillers	2023	Yutong Bai Zeyu Wang Junfei Xiao Wei Chen Huiyu Wang Alan Yuille Yuyin Zhou Cihang Xie
+ PDF Chat	DOT: A Distillation-Oriented Trainer	2023	Borui Zhao Quan Cui Renjie Song Jiajun Liang

Works That Cite This (31)

Action	Title	Year	Authors
+ PDF Chat	Dynamic Slimmable Network	2021	Changlin Li Guangrun Wang Bing Wang Xiaodan Liang Zhihui Li Xiaojun Chang
+ PDF Chat	Joint Multi-Dimension Pruning via Numerical Gradient Update	2021	Zechun Liu Xiangyu Zhang Zhiqiang Shen Yichen Wei Kwang‐Ting Cheng Jian Sun
+ PDF Chat	Toward Understanding and Boosting Adversarial Transferability From a Distribution Perspective	2022	Yao Zhu Yuefeng Chen Xiaodan Li Kejiang Chen Yuan He Xiang Tian Bolun Zheng Yaowu Chen Qingming Huang
+	Arch-Net: Model Distillation for Architecture Agnostic Model Deployment	2021	Weixin Xu Zipeng Feng Shuangkang Fang Song Yuan Yi Yang Shuchang Zhou
+ PDF Chat	MobileOne: An Improved One millisecond Mobile Backbone	2023	Pavan Kumar Anasosalu Vasu James Gabriel Jeff Zhu Oncel Tuzel Anurag Ranjan
+	S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration	2021	Zhiqiang Shen Zechun Liu Jie Qin Lei Huang Kwang‐Ting Cheng Marios Savvides
+ PDF Chat	S<sup>2</sup>-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration	2021	Zhiqiang Shen Zechun Liu Jie Qin Lei Huang Kwang‐Ting Cheng Marios Savvides
+	6.7ms on Mobile with over 78% ImageNet Accuracy: Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration.	2020	Zhengang Li Geng Yuan Wei Niu Yanyu Li Pu Zhao Yuxuan Cai Xuan Shen Zheng Zhan Zhenglun Kong Qing Jin
+	NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration	2020	Zhengang Li Geng Yuan Wei Niu Pu Zhao Yanyu Li Yuxuan Cai Xuan Shen Zheng Zhan Zhenglun Kong Qing Jin
+	Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study	2021	Zhiqiang Shen Zechun Liu Dejia Xu Zitian Chen Kwang‐Ting Cheng Marios Savvides

Works Cited by This (39)

Action	Title	Year	Authors
+	FitNets: Hints for Thin Deep Nets	2014	Adriana Romero Nicolas Ballas Samira Ebrahimi Kahou Antoine Chassang Carlo Gatta Yoshua Bengio
+	Distilling the Knowledge in a Neural Network	2015	Geoffrey E. Hinton Oriol Vinyals Jay B. Dean
+ PDF Chat	Fully convolutional networks for semantic segmentation	2015	Jonathan Long Evan Shelhamer Trevor Darrell
+ PDF Chat	Going deeper with convolutions	2015	Christian Szegedy Wei Liu Yangqing Jia Pierre Sermanet Scott Reed Dragomir Anguelov Dumitru Erhan Vincent Vanhoucke Andrew Rabinovich
+ PDF Chat	Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation	2014	Ross Girshick Jeff Donahue Trevor Darrell Jitendra Malik
+ PDF Chat	Rethinking the Inception Architecture for Computer Vision	2016	Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jon Shlens Zbigniew Wojna
+ PDF Chat	Deep Residual Learning for Image Recognition	2016	Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
+ PDF Chat	XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks	2016	Mohammad Rastegari Vicente Ordóñez Joseph Redmon Ali Farhadi
+	Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1	2016	Matthieu Courbariaux Itay Hubara Daniel Soudry Ran El‐Yaniv Yoshua Bengio
+	Pruning Filters for Efficient ConvNets	2016	Hao Li Asim Kadav Igor Đurđanović Hanan Samet Hans Peter Graf