Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networks

Leni Ven, Johannes Lederer

Type: Preprint

Publication Date: 2021-01-01

Citations: 2

DOI: https://doi.org/10.48550/arxiv.2106.02260

View Publication

Locations

arXiv (Cornell University) - View - PDF
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	Revise Saturated Activation Functions	2016	Bing Xu Ruitong Huang Mu Li
+	Gradients explode - Deep Networks are shallow - ResNet explained	2017	George Philipp Dawn Song Jaime Carbonell
+ PDF Chat	Adaptively Customizing Activation Functions for Various Layers	2022	Haigen Hu Aizhu Liu Qiu Guan Hanwang Qian Xiaoxin Li Shengyong Chen Qianwei Zhou
+	Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping	2021	James Martens Andy Ballard Guillaume Desjardins Grzegorz Świrszcz Valentin Dalibard Jascha Sohl‐Dickstein Samuel S. Schoenholz
+ PDF Chat	Stable and Robust Deep Learning By Hyperbolic Tangent Exponential Linear Unit (TeLU)	2024	Alfredo Fernandez Ankur Mali
+	Adaptively Customizing Activation Functions for Various Layers	2021	Haigen Hu Aizhu Liu Qiu Guan Xiaoxin Li Shengyong Chen Qianwei Zhou
+	Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping.	2021	James Martens Andy Ballard Guillaume Desjardins Grzegorz Świrszcz Valentin Dalibard Jascha Sohl‐Dickstein Samuel S. Schoenholz
+	Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks	2021	Mathew Mithra Noel L Arunkumar Advait Trivedi Praneet Dutta
+	Activation Functions in Artificial Neural Networks: A Systematic Overview	2021	Johannes Lederer
+	MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes	2018	Henry Gouk Bernhard Pfahringer Eibe Frank Michael J. Cree
+	MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes	2018	Henry Gouk Bernhard Pfahringer Eibe Frank Michael J. Cree
+	A Methodology for Automatic Selection of Activation Functions to Design Hybrid Deep Neural Networks	2018	Alberto Marchisio Muhammad Abdullah Hanif Semeen Rehman Maurizio Martina Muhammad Shafique
+	Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks	2018	Leon René Sütfeld Flemming Brieger Holger Finger Sonja Füllhase Gordon Pipa
+	Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks	2018	Leon René Sütfeld Flemming Brieger Holger Finger Sonja Füllhase Gordon Pipa
+	Normalized Activation Function: Toward Better Convergence	2022	Yuan Peiwen Changsheng Zhu
+	Smooth activations and reproducibility in deep networks	2020	Gil I. Shamir Dong Lin Lorenzo Coviello
+	LiSHT: Non-Parametric Linearly Scaled Hyperbolic Tangent Activation Function for Neural Networks	2019	Swalpa Kumar Roy Suvojit Manna Shiv Ram Dubey B.B. Chaudhuri
+	Fundamental differences between Dropout and Weight Decay in Deep Networks	2016	David P. Helmbold Philip M. Long
+ PDF Chat	Deep Learning Activation Functions: Fixed-Shape, Parametric, Adaptive, Stochastic, Miscellaneous, Non-Standard, Ensemble	2024	Mahmoud Hammad
+ PDF Chat	Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks	2024	Vladimír Kunc Jǐŕı Kléma

Works That Cite This (0)

Action	Title	Year	Authors

Works Cited by This (0)

Action	Title	Year	Authors