On the importance of single directions for generalization

Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, Matthew Botvinick

Type: Preprint

Publication Date: 2018-01-01

Citations: 202

DOI: https://doi.org/10.48550/arxiv.1803.06959

View Publication

Locations

arXiv (Cornell University) - View
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	On the importance of single directions for generalization	2018	Ari S. Morcos David G. T. Barrett Neil C. Rabinowitz Matthew Botvinick
+	Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale	2020	Piotr Zieliński Shankar Krishnan Satrajit Chatterjee
+ PDF Chat	A separability-based approach to quantifying generalization: which layer is best?	2024	Luciano Dyballa Evan Gerritz Steven W. Zucker
+	Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning	2022	Florian Bordes Randall Balestriero Quentin Garrido Adrien Bardes P. Vincent
+	Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale	2021	Piotr Zieliński Shankar Krishnan Satrajit Chatterjee
+ PDF Chat	Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data	2024	Nikita Tsoy Nikola Konstantinov
+ PDF Chat	Deep Neural Network Models Trained With A Fixed Random Classifier Transfer Better Across Domains	2024	Hafiz Tiomoko Ali Umberto Michieli Ji Joong Moon Daehyun Kim Mete Özay
+	Deep Neural Network Models Trained with a Fixed Random Classifier Transfer Better Across Domains	2024	Hafiz Tiomoko Ali Umberto Michieli Ji Joong Moon Daehyun Kim Mete Özay
+	Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift	2015	Sergey Ioffe Christian Szegedy
+	Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift	2015	Sergey Ioffe Christian Szegedy
+	Accelerating Training of Deep Neural Networks with a Standardization Loss	2019	Jasmine Collins Johannes Ballé Jonathon Shlens
+	Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning	2021	Ekdeep Singh Lubana Robert P. Dick Hidenori Tanaka
+ PDF Chat	What Variables Affect Out-Of-Distribution Generalization in Pretrained Models?	2024	Md Yousuf Harun Kyungbok Lee Jhair Gallardo Giri P. Krishnan Christopher Kanan
+ PDF Chat	Feature Contamination: Neural Networks Learn Uncorrelated Features and Fail to Generalize	2024	Tianren Zhang Chujie Zhao Guanyu Chen Yizhou Jiang Chen Feng
+	No Reason for No Supervision: Improved Generalization in Supervised Models	2022	Mert Bülent Sarıyıldız Yannis Kalantidis Karteek Alahari Diane Larlus
+	Neuron with Steady Response Leads to Better Generalization	2021	Qiang Fu Lun Du Haitao Mao Chen Xu Wei Fang Shi Han Dongmei Zhang
+	Training Deep Networks from Zero to Hero: avoiding pitfalls and going beyond	2021	Moacir Antonelli Ponti Fernando Pereira dos Santos Leo Sampaio Ferraz Ribeiro Gabriel B. Cavallari
+	Generalization in Neural Networks: A Broad Survey	2022	Chris Rohlfs
+	Beyond BatchNorm: Towards a General Understanding of Normalization in Deep Learning	2021	Ekdeep Singh Lubana Robert P. Dick Hidenori Tanaka
+	SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization	2021	Soroosh Shahtalebi Jean-Christophe Gagnon-Audet Touraj Laleh Mojtaba Faramarzi Kartik Ahuja Irina Rish

Works That Cite This (110)

Action	Title	Year	Authors
+	Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey	2020	Lei Deng Guoqi Li Song Han Luping Shi Yuan Xie
+ PDF Chat	The roles of supervised machine learning in systems neuroscience	2019	Joshua I. Glaser Ari S. Benjamin Roozbeh Farhoodi Konrad P. Körding
+	Rethinking Normalization and Elimination Singularity in Neural Networks	2019	Siyuan Qiao Huiyu Wang Chenxi Liu Wei Shen Alan Yuille
+	CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations	2021	Leila Arras Ahmed Osman Wojciech Samek
+	Representation Based Complexity Measures for Predicting Generalization in Deep Learning	2020	Parth Natekar Manik Sharma
+	Towards Ground Truth Evaluation of Visual Explanations	2020	Ahmed Osman Leila Arras Wojciech Samek
+	An Efficient Explorative Sampling Considering the Generative Boundaries of Deep Generative Neural Networks	2019	Giyoung Jeon Haedong Jeong Jaesik Choi
+	On the Margin Theory of Feedforward Neural Networks	2018	Colin Wei Jason D. Lee Qiang Liu Tengyu Ma
+	Learned Deformation Stability in Convolutional Neural Networks.	2018	Avraham Ruderman Neil C. Rabinowitz Ari S. Morcos Daniel Zoran
+	NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks.	2018	Fahim Dalvi Avery Nortonsmith David Bau Yonatan Belinkov Hassan Sajjad Nadir Durrani James Glass

Works Cited by This (8)

Action	Title	Year	Authors
+ PDF Chat	Deep Residual Learning for Image Recognition	2016	Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
+	Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation	2016	Yonghui Wu Mike Schuster Zhifeng Chen Quoc V. Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey
+	Opening the Black Box of Deep Neural Networks via Information	2017	Ravid Shwartz-Ziv Naftali Tishby
+	Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data	2017	Gintare Karolina Dziugaite Daniel M. Roy
+	On the Emergence of Invariance and Disentangling in Deep Representations.	2017	Alessandro Achille Stefano Soatto
+	Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift	2015	Sergey Ioffe Christian Szegedy
+ PDF Chat	Network Dissection: Quantifying Interpretability of Deep Visual Representations	2017	David Bau Bolei Zhou Aditya Khosla Aude Oliva Antonio Torralba
+	On the Expressive Power of Deep Neural Networks	2016	Maithra Raghu Ben Poole Jon Kleinberg Surya Ganguli Jascha Sohl‐Dickstein