Smooth activations and reproducibility in deep networks

Gil I. Shamir, Dong Lin, Lorenzo Coviello

Type: Preprint

Publication Date: 2020-01-01

Citations: 7

DOI: https://doi.org/10.48550/arxiv.2010.09931

View Publication

Locations

arXiv (Cornell University) - View
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)	2015	Djork-Arné Clevert Thomas Unterthiner Sepp Hochreiter
+	Piecewise Linear Units Improve Deep Neural Networks	2021	Jordan Inturrisi Suiyang Khoo Abbas Z. Kouzani Riccardo M. Pagliarella
+	When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?	2021	Niladri S. Chatterji Philip M. Long Peter L. Bartlett
+	Gradients explode - Deep Networks are shallow - ResNet explained	2017	George Philipp Dawn Song Jaime Carbonell
+ PDF Chat	Hysteresis Activation Function for Efficient Inference	2024	Moshe Kimhi Idan Kashani Avi Mendelson Chaim Baskin
+	Householder-Absolute Neural Layers For High Variability and Deep Trainability.	2021	Yueyao Yu Yin Zhang⋆
+	E-swish: Adjusting Activations to Different Network Depths	2018	Eric Alcaide
+	Farkas layers: don't shift the data, fix the geometry.	2019	Aram-Alexandre Pooladian Chris Finlay Adam M. Oberman
+	E-swish: Adjusting Activations to Different Network Depths.	2018	Eric Alcaide
+	Farkas layers: don't shift the data, fix the geometry	2019	Aram-Alexandre Pooladian Christopher C. Finlay Adam M. Oberman
+	Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks	2018	Leon René Sütfeld Flemming Brieger Holger Finger Sonja Füllhase Gordon Pipa
+	Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks	2018	Leon René Sütfeld Flemming Brieger Holger Finger Sonja Füllhase Gordon Pipa
+ PDF Chat	Stable and Robust Deep Learning By Hyperbolic Tangent Exponential Linear Unit (TeLU)	2024	Alfredo Fernandez Ankur Mali
+	FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks	2017	Suo Qiu Xiangmin Xu Bolun Cai
+	Flexible Rectified Linear Units for Improving Convolutional Neural Networks	2017	Suo Qiu Bolun Cai
+	Empirical Evaluation of Rectified Activations in Convolutional Network	2015	Bing Xu Naiyan Wang Tianqi Chen Mu Li
+	Rational neural networks	2020	Nicolas Boullé Yuji Nakatsukasa Alex Townsend
+	Rational neural networks	2020	Nicolas Boullé Yuji Nakatsukasa Alex Townsend
+	Learning specialized activation functions with the Piecewise Linear Unit	2021	Yucong Zhou Zezhou Zhu Zhao Zhong
+	Data-aware customization of activation functions reduces neural network error	2023	Fuchang Gao Boyu Zhang

Works That Cite This (4)

Action	Title	Year	Authors
+	Smooth Adversarial Training	2020	Cihang Xie Mingxing Tan Boqing Gong Alan Yuille Quoc V. Le
+	Randomness In Neural Network Training: Characterizing The Impact of Tooling	2021	Donglin Zhuang Xingyao Zhang Shuaiwen Leon Song Sara Hooker
+	Synthesizing Irreproducibility in Deep Networks.	2021	Robert R. Snapp Gil I. Shamir
+	Dropout Prediction Variation Estimation Using Neuron Activation Strength.	2021	Haichao Yu Zhe Chen Dong Lin Gil I. Shamir Jie Han

Works Cited by This (23)

Action	Title	Year	Authors
+	Distilling the Knowledge in a Neural Network	2015	Geoffrey E. Hinton Oriol Vinyals Jay B. Dean
+ PDF Chat	Robust Estimation of a Location Parameter	1964	Peter J. Huber
+	On the Number of Linear Regions of Deep Neural Networks	2014	Guido Montúfar Razvan Pascanu Kyunghyun Cho Yoshua Bengio
+	Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)	2015	Djork-Arné Clevert Thomas Unterthiner Sepp Hochreiter
+	Continuously Differentiable Exponential Linear Units	2017	Jonathan T. Barron
+	Critical Learning Periods in Deep Neural Networks	2017	Alessandro Achille Matteo Rovere Stefano Soatto
+	Large scale distributed neural network training through online distillation	2018	Rohan Anil Gabriel Pereyra Alexandre Passos Róbert Ormándi George E. Dahl Geoffrey E. Hinton
+	Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning	2018	Zhi Chen Pin–Han Ho
+	Deterministic Implementations for Reproducibility in Deep Reinforcement Learning	2018	Prabhat Nagarajan Garrett Warnell Peter Stone
+	Gaussian Error Linear Units (GELUs)	2016	Dan Hendrycks Kevin Gimpel