Memory capacity of neural networks with threshold and ReLU activations

Roman Vershynin

Type: Preprint

Publication Date: 2020-01-01

Citations: 13

DOI: https://doi.org/10.48550/arxiv.2001.06938

View Publication

Locations

arXiv (Cornell University) - View
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	Network size and weights size for memorization with two-layers neural networks	2020	Sébastien Bubeck Ronen Eldan Yin Tat Lee Dan Mikulincer
+ PDF Chat	Neural networks with linear threshold activations: structure and algorithms	2023	Sammy Khalife Hongyu Cheng Amitabh Basu
+ PDF Chat	Neural Networks with Linear Threshold Activations: Structure and Algorithms	2022	Sammy Khalife Amitabh Basu
+	The capacity of feedforward neural networks	2019	Pierre Baldi Roman Vershynin
+	The capacity of feedforward neural networks	2019	Pierre Baldi Roman Vershynin
+	Neural networks with linear threshold activations: structure and algorithms	2021	Sammy Khalife Hongyu Cheng Amitabh Basu
+ PDF Chat	Fixed width treelike neural networks capacity analysis -- generic activations	2024	Mihailo Stojnic
+	Hidden unit specialization in layered neural networks: ReLU vs. sigmoidal activation	2020	Elisa Oostwal Michiel Straat Michael Biehl
+	Memory capacity of two layer neural networks with smooth activations	2023	Liam Madden Christos Thrampoulidis
+	Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently	2019	Rong Ge Runzhe Wang Haoyu Zhao
+ PDF Chat	Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks	2024	Hemanth Saratchandran Shin-Fang Chng Simon Lucey
+	Memory Capacity of Two Layer Neural Networks with Smooth Activations	2024	Liam Madden Christos Thrampoulidis
+ PDF Chat	Learning Neural Networks with Sparse Activations	2024	Pranjal Awasthi Nishanth Dikkala Pritish Kamath Raghu Meka
+	The capacity of feedforward neural networks	2019	Pierre Baldi Roman Vershynin
+	On Sparsity in Overparametrised Shallow ReLU Networks.	2020	Jaume de Dios Joan Bruna
+ PDF Chat	Properties of the Geometry of Solutions and Capacity of Multilayer Neural Networks with Rectified Linear Unit Activations	2019	Carlo Baldassi Enrico M. Malatesta Riccardo Zecchina
+	On Sparsity in Overparametrised Shallow ReLU Networks	2020	Jaume de Dios Joan Bruna
+	The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers	2022	Zonglin Li Chong You Srinadh Bhojanapalli Daliang Li Ankit Singh Rawat Sashank J. Reddi Ke Ye Felix Chern Felix Yu Ruiqi Guo
+	A Capacity Scaling Law for Artificial Neural Networks	2017	Gerald Friedland Mario Michael Krell
+	Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs	2023	Dmitry Chistikov Matthias Englert Ranko Lazić

Works That Cite This (10)

Action	Title	Year	Authors
+	Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory	2020	Paulo Tabuada Bahman Gharesifard
+	Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology	2020	Quynh Nguyen Marco Mondelli
+ PDF Chat	Approximation in shift-invariant spaces with deep ReLU neural networks	2022	Yunfei Yang Zhen Li Yang Wang
+ PDF Chat	Provable Memorization via Deep Neural Networks using Sub-linear Parameters	2020	Sejun Park Jaeho Lee Chulhee Yun Jinwoo Shin
+ PDF Chat	The interpolation phase transition in neural networks: Memorization and generalization under lazy training	2022	Andrea Montanari Yiqiao Zhong
+ PDF Chat	Random vector functional link networks for function approximation on manifolds	2024	Deanna Needell Aaron A. Nelson Rayan Saab Palina Salanevich Olov Schavemaker
+	On the Optimal Memorization Power of ReLU Neural Networks	2021	Gal Vardi Gilad Yehudai Ohad Shamir
+	Memory Capacity of Neural Turing Machines with Matrix Representation.	2021	Animesh Renanse Rohitash Chandra Alok Sharma
+	Minimum Width for Universal Approximation	2020	Sejun Park Chulhee Yun Jaeho Lee Jinwoo Shin
+	Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers	2021	Colin Wei Yining Chen Tengyu Ma

Works Cited by This (14)

Action	Title	Year	Authors
+	Gradient Descent Provably Optimizes Over-parameterized Neural Networks	2018	Simon S. Du Xiyu Zhai Barnabás Póczos Aarti Singh
+	A Convergence Theory for Deep Learning via Over-Parameterization	2018	Zeyuan Allen-Zhu Yuanzhi Li Zhao Song
+	Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks	2018	Difan Zou Yuan Cao Dongruo Zhou Quanquan Gu
+	Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks	2019	Sanjeev Arora Simon S. Du Wei Hu Zhiyuan Li Ruosong Wang
+	Towards moderate overparameterization: global convergence guarantees for training shallow neural networks	2019	Samet Oymak Mahdi Soltanolkotabi
+	Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound	2019	Zhao Song Xin Yang
+	An Improved Analysis of Training Over-parameterized Deep Neural Networks	2019	Difan Zou Quanquan Gu
+	Identity Matters in Deep Learning	2016	Moritz Hardt Tengyu Ma
+	The capacity of feedforward neural networks	2019	Pierre Baldi Roman Vershynin
+	Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity	2019	Chulhee Yun Suvrit Sra Ali Jadbabaie