Understanding the Limitations of Variational Mutual Information Estimators

Jiaming Song, Stefano Ermon

Type: Preprint

Publication Date: 2019-01-01

Citations: 98

DOI: https://doi.org/10.48550/arxiv.1910.06222

View

Locations

arXiv (Cornell University) - View - PDF
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	Understanding the Limitations of Variational Mutual Information Estimators	2019	Jiaming Song Stefano Ermon
+	Data-Efficient Mutual Information Neural Estimator	2019	Lin Xiao Indranil Sur Samuel A. Nastase Ajay Divakaran Uri Hasson Mohamed R. Amer
+	Data-Efficient Mutual Information Neural Estimator	2019	Lin Xiao Indranil Sur Samuel A. Nastase Ajay Divakaran Uri Hasson Mohamed R. Amer
+	On Variational Bounds of Mutual Information	2019	Ben Poole Sherjil Ozair Aäron van den Oord Alexander A. Alemi George Tucker
+	Regularized Mutual Information Neural Estimation	2021	Kwanghee Choi Siyeong Lee
+	The Role of Mutual Information in Variational Classifiers	2020	Matías Vera Leonardo Rey Vega Pablo Piantanida
+	DEMI: Discriminative Estimator of Mutual Information	2020	Ruizhi Liao Daniel Moyer Polina Golland William M. Wells
+	Variational $f$-Divergence and Derangements for Discriminative Mutual Information Estimation	2023	Nunzio A. Letizia Nicola Novello Andrea M. Tonello
+	Mutual Information Constraints for Monte-Carlo Objectives	2020	Gábor Melis András György Phil Blunsom
+	Mutual Information Constraints for Monte-Carlo Objectives.	2020	Gábor Melis András György Phil Blunsom
+	Reducing the Variance of Variational Estimates of Mutual Information by Limiting the Critic's Hypothesis Space to RKHS	2020	P Aditya Sreekar Ujjwal Tiwari Anoop Namboodiri
+	Reducing the Variance of Variational Estimates of Mutual Information by Limiting the Critic's Hypothesis Space to RKHS	2020	P Aditya Sreekar Ujjwal Tiwari Anoop Namboodiri
+	InfoNCE is a variational autoencoder.	2021	Laurence Aitchison
+	VMI-VAE: Variational Mutual Information Maximization Framework for VAE With Discrete and Continuous Priors	2020	Andriy Serdega Dae‐Shik Kim
+ PDF Chat	Learning Representations by Maximizing Mutual Information in Variational Autoencoders	2020	Ali Lotfi Rezaabad Sriram Vishwanath
+	Learning Representations by Maximizing Mutual Information in Variational Autoencoders	2019	Ali Lotfi Rezaabad Sriram Vishwanath
+	Learning Representations by Maximizing Mutual Information in Variational Autoencoders	2019	Ali Lotfi Rezaabad Sriram Vishwanath
+ PDF Chat	Reducing the Variance of Variational Estimates of Mutual Information by Limiting the Critic's Hypothesis Space to RKHS	2021	P Aditya Sreekar Ujjwal Tiwari Anoop Namboodiri
+	Neural Entropic Estimation: A faster path to mutual information estimation	2019	Chung Chan Ali Al-Bashabsheh Hing Pang Huang Michael Lim Da Sun Handason Tam Chao Zhao
+	Max-Sliced Mutual Information	2023	Dor Tsur Ziv Goldfeld Kristjan Greenewald

Cited by (74)

Action	Title	Year	Authors
+	Density Ratio Estimation via Infinitesimal Classification	2021	Kristy Choi Chenlin Meng Yang Song Stefano Ermon
+	Learning Conditional Invariance through Cycle Consistency	2021	Maxim Samarin Vitali Nesterov Mario Wieser Aleksander Wieczorek Sonali Parbhoo Volker Röth
+ PDF Chat	Multimodal Representation Learning via Maximization of Local Mutual Information	2021	Ruizhi Liao Daniel Moyer Miriam Cha Keegan Quigley Seth J. Berkowitz Steven Horng Polina Golland William M. Wells
+ PDF Chat	Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization	2021	L. Zhao Yuxiao Wang Jiaping Zhao Liangzhe Yuan Jennifer J. Sun Florian Schroff Hartwig Adam Xi Peng Dimitris Metaxas Ting Liu
+ PDF Chat	Learning Conditional Invariance Through Cycle Consistency	2021	Maxim Samarin Vitali Nesterov Mario Wieser Aleksander Wieczorek Sonali Parbhoo Volker Röth
+	No MCMC for me: Amortized sampling for fast and stable training of energy-based models	2021	Will Grathwohl Jacob Kelly Milad Hashemi Mohammad Norouzi Kevin Swersky David Duvenaud
+	Discriminative Mutual Information Estimators for Channel Capacity Learning	2021	Nunzio A. Letizia Andrea M. Tonello
+ PDF Chat	Parametric Copula-GP model for analyzing multidimensional neuronal and behavioral relationships	2022	Nina Kudryashova Theoklitos Amvrosiadis Nathalie Dupuy Nathalie L. Rochefort Arno Onken
+	A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification	2023	Pierre Colombo Nathan Noiry Guillaume Staerman Pablo Piantanida
+ PDF Chat	MAPPING: Debiasing Graph Neural Networks for Fair Node Classification with Limited Sensitive Information Leakage	2024	Ying Song Balaji Palanisamy
+	A Novel Information Theoretic Objective to Disentangle Representations for Fair Classification	2023	Pierre Colombo Nathan Noiry Guillaume Staerman Pablo Piantanida
+	DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation	2021	Alexandre Ramé Matthieu Cord
+	Neural Mutual Information Estimation for Channel Coding: State-of-the-Art Estimators, Analysis, and Performance Comparison	2020	Rick Fritschek Rafael F. Schaefer Gerhard Wunder
+	AstroCLIP: Cross-Modal Pre-Training for Astronomical Foundation Models	2023	François Lanusse Liam Parker Siavash Golkar Miles Cranmer Alberto Bietti Michael Eickenberg Géraud Krawezik Michael T. McCabe Ruben Ohana M. Pettee
+	C-MI-GAN : Estimation of Conditional Mutual Information using MinMax formulation	2020	Arnab Kumar Mondal Arnab Bhattacharya Sudipto Mukherjee Prathosh AP Sreeram Kannan Himanshu Asnani
+ PDF Chat	AstroCLIP: A cross-modal foundation model for galaxies	2024	Liam Parker François Lanusse Siavash Golkar Leopoldo Sarra Miles Cranmer Alberto Bietti Michael Eickenberg Géraud Krawezik Michael T. McCabe Rudy Morel
+ PDF Chat	Learning Disentangled Textual Representations via Statistical Measures of Similarity	2022	Pierre Colombo Guillaume Staerman Nathan Noiry Pablo Piantanida
+ PDF Chat	A Reverse Jensen Inequality Result with Application to Mutual Information Estimation	2021	Gerhard Wunder Benedikt Gros Rick Fritschek Rafael F. Schaefer
+ PDF Chat	InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle	2022	Mengting Xu Tao Zhang Zhongnian Li Daoqiang Zhang
+	Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE	2021	Junya Chen Zhe Gan Xuan Li Qing Guo Li‐Qun Chen Shuyang Gao Tagyoung Chung Yi Xu Belinda Zeng Wenlian Lu
+	Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization	2021	Jizong Peng Marco Pedersoli Christian Desrosiers
+ PDF Chat	Disentangled Information Bottleneck	2021	Ziqi Pan Li Niu Jianfu Zhang Liqing Zhang
+ PDF Chat	Neural Mutual Information Estimation for Channel Coding: State-of-the-Art Estimators, Analysis, and Performance Comparison	2020	Rick Fritschek Rafael F. Schaefer Gerhard Wunder
+ PDF Chat	Neural Estimation and Optimization of Directed Information Over Continuous Spaces	2023	Dor Tsur Ziv Aharoni Ziv Goldfeld Haim H. Permuter
+ PDF Chat	Neural Capacity Estimators: How Reliable Are They?	2022	Farhad Mirkarimi Stefano Rini Nariman Farsad
+ PDF Chat	AID-Purifier: A Light Auxiliary Network for Boosting Adversarial Defense	2022	Duhun Hwang Eunjung Lee Wonjong Rhee
+	Self-supervised Learning from a Multi-view Perspective	2020	Yao-Hung Hubert Tsai Yue Wu Ruslan Salakhutdinov Louis‐Philippe Morency
+ PDF Chat	Disentangling Label Distribution for Long-tailed Visual Recognition	2021	Youngkyu Hong Seungju Han Kwanghee Choi Seokjun Seo Beomsu Kim Buru Chang
+	Improving Multimodal fusion via Mutual Dependency Maximisation	2021	Pierre Colombo Émile Chapuis Matthieu Labeau Chloé Clavel
+ PDF Chat	Bottleneck Problems: An Information and Estimation-Theoretic View	2020	Shahab Asoodeh Flávio P. Calmon
+ PDF Chat	MICAL: Mutual Information-Based CNN-Aided Learned Factor Graphs for Seizure Detection From EEG Signals	2023	Bahareh Salafian Eyal Fishel Ben-Knaan Nir Shlezinger Sandrine de Ribaupierre Nariman Farsad
+	InfoShape: Task-Based Neural Data Shaping via Mutual Information	2023	Homa Esfahanizadeh William K.K. Wu Manya Ghobadi Regina Barzilay Muriel Médard
+	No MCMC for me: Amortized sampling for fast and stable training of energy-based models.	2020	Will Grathwohl Jacob Kelly Milad Hashemi Mohammad Norouzi Kevin Swersky David Duvenaud
+ PDF Chat	Contrastive Learning for Weakly Supervised Phrase Grounding	2020	Tanmay Gupta Arash Vahdat Gal Chechik Xiaodong Yang Jan Kautz Derek Hoiem
+	Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization	2020	L. Zhao Yuxiao Wang Jiaping Zhao Liangzhe Yuan Jennifer J. Sun Florian Schroff Hartwig Adam Xi Peng Dimitris Metaxas Ting Liu
+	Disentangling Label Distribution for Long-tailed Visual Recognition	2020	Youngkyu Hong Seungju Han Kwanghee Choi Seokjun Seo Beomsu Kim Buru Chang
+	Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness	2020	L. Zhao Ting Liu Xi Peng Dimitris Metaxas
+	Cross-Modal Contrastive Learning for Text-to-Image Generation	2021	Han Zhang Jing Yu Koh Jason Baldridge Honglak Lee Yinfei Yang
+	A Reverse Jensen Inequality Result with Application to Mutual Information Estimation.	2021	Gerhard Wunder Benedikt Groß Rick Fritschek Rafael F. Schaefer
+	Contrastive Learning for Weakly Supervised Phrase Grounding	2020	Tanmay Gupta Arash Vahdat Gal Chechik Xiaodong Yang Jan Kautz Derek Hoiem

Citing (24)

Action	Title	Year	Authors
+	NICE: Non-linear Independent Components Estimation	2014	Laurent Dinh David Krueger Yoshua Bengio
+ PDF Chat	Entropy and information in neural spike trains: Progress on the sampling problem	2004	Ilya Nemenman William Bialek Rob de Ruyter van Steveninck
+	Asymptotic evaluation of certain markov process expectations for large time, II	1975	M. D. Donsker S. R. S. Varadhan
+ PDF Chat	Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization	2010	XuanLong Nguyen Martin J. Wainwright Michael I. Jordan
+	Density estimation using Real NVP	2016	Laurent Dinh Jascha Sohl‐Dickstein Samy Bengio
+	Deep Variational Information Bottleneck	2016	Alexander A. Alemi Ian Fischer Joshua V. Dillon Kevin Murphy
+ PDF Chat	Boosted Generative Models	2018	Aditya Grover Stefano Ermon
+	MINE: Mutual Information Neural Estimation.	2018	Ishmael Belghazi Sai Rajeswar Aristide Baratin R Devon Hjelm Aaron Courville
+	The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models	2018	Shengjia Zhao Jiaming Song Stefano Ermon
+	Representation Learning with Contrastive Predictive Coding	2018	Aäron van den Oord Yazhe Li Oriol Vinyals
+	Formal Limitations on the Measurement of Mutual Information	2018	David McAllester Karl Stratos
+	Towards a Definition of Disentangled Representations	2018	Irina Higgins David Amos David Pfau Sébastien Racanière Löıc Matthey Danilo Jimenez Rezende Alexander Lerchner
+	On Variational Bounds of Mutual Information	2019	Ben Poole Sherjil Ozair Aäron van den Oord Alexander A. Alemi George Tucker
+	Learning deep representations by mutual information estimation and maximization	2018	R Devon Hjelm Alex Fedorov Samuel Lavoie-Marchildon Karan Grewal Phil Bachman Adam Trischler Yoshua Bengio
+	Efficient Estimation of Mutual Information for Strongly Dependent Variables	2014	Shuyang Gao Greg Ver Steeg Aram Galstyan
+	Estimating Mutual Information for Discrete-Continuous Mixtures	2017	Weihao Gao Sreeram Kannan Sewoong Oh Pramod Viswanath
+	InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets	2016	Xi Chen Yan Duan Rein Houthooft John Schulman Ilya Sutskever Pieter Abbeel
+	Bias and Generalization in Deep Generative Models: An Empirical Study	2018	Shengjia Zhao Hongyu Ren Arianna Yuan Jiaming Song Noah D. Goodman Stefano Ermon
+ PDF Chat	Curiosity-Driven Exploration by Self-Supervised Prediction	2017	Deepak Pathak Pulkit Agrawal Alexei A. Efros Trevor Darrell
+	f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization	2016	Sebastian Nowozin Botond Cseke Ryota Tomioka
+	Adam: A Method for Stochastic Optimization	2014	Diederik P. Kingma Jimmy Ba
+ PDF Chat	Deep learning and the information bottleneck principle	2015	Naftali Tishby Noga Zaslavsky
+	On Mutual Information Maximization for Representation Learning	2019	Michael Tschannen Josip Djolonga Paul K. Rubenstein Sylvain Gelly Mario Lučić
+	CCMI : Classifier based Conditional Mutual Information Estimation	2019	Sudipto Mukherjee Himanshu Asnani Sreeram Kannan