Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity

Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, Lav R. Varshney

Type: Preprint

Publication Date: 2020-01-01

Citations: 11

DOI: https://doi.org/10.48550/arxiv.2007.14966

View Publication

Locations

arXiv (Cornell University) - View
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	Mirostat: A Perplexity-Controlled Neural Text Decoding Algorithm	2020	Sourya Basu Govardana Sachitanandam Ramachandran Nitish Shirish Keskar Lav R. Varshney
+	Truncation Sampling as Language Model Desmoothing	2022	John K. Hewitt Christopher D. Manning Percy Liang
+	On the Efficacy of Sampling Adapters	2023	Clara Meister Tiago Pimentel L. Malagutti Ethan Wilcox Ryan Cotterell
+	Contrastive Decoding: Open-ended Text Generation as Optimization	2022	Xiang Lisa Li Ari Holtzman Daniel Fried Percy Liang Jason Eisner Tatsunori Hashimoto Luke Zettlemoyer Michael Lewis
+	Truncation Sampling as Language Model Desmoothing	2022	John K. Hewitt Christopher D. Manning Percy Liang
+ PDF Chat	Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding	2023	Mirac Süzgün Luke Melas-Kyriazi Dan Jurafsky
+ PDF Chat	mbrs: A Library for Minimum Bayes Risk Decoding	2024	Hiroyuki Deguchi Yusuke Sakai Hidetaka Kamigaito Taro Watanabe
+ PDF Chat	Min P Sampling: Balancing Creativity and Coherence at High Temperature	2024	Minh‐Hoang Nguyen Andrew M. Baker Andreas Kirsch Clement Neo
+	The Curious Case of Neural Text Degeneration	2020	Ari Holtzman Jan Buys Leo Du Maxwell Forbes Yejin Choi
+	The Curious Case of Neural Text Degeneration	2019	Ari Holtzman Jan Buys Li Du Maxwell Forbes Yejin Choi
+ PDF Chat	Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation	2024	Esteban Garces Arias Julian Rodemann Meimingwei Li Christian Heumann Matthias Aßenmacher
+	On the probability–quality paradox in language generation	2022	Clara Meister Gian Wiher Tiago Pimentel Ryan Cotterell
+	On the probability-quality paradox in language generation	2022	Clara Meister Gian Wiher Tiago Pimentel Ryan Cotterell
+	Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding	2022	Mirac Süzgün Luke Melas-Kyriazi Dan Jurafsky
+	Closing the Curious Case of Neural Text Degeneration	2023	Matthew Finlayson John K. Hewitt Alexander Koller Swabha Swayamdipta Ashish Sabharwal
+ PDF Chat	Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation	2024	Yuxuan Zhou Margret Keuper Mario Fritz
+	Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation	2023	Markus Freitag Behrooz Ghorbani Patrick Fernandes
+	Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation	2023	Markus Freitag Behrooz Ghorbani Patrick Fernandes
+ PDF Chat	REAL Sampling: Boosting Factuality and Diversity of Open-Ended Generation via Asymptotic Entropy	2024	Haw-Shiuan Chang Nanyun Peng Mohit Bansal Anil Ramakrishna Tagyoung Chung
+ PDF Chat	Locally Typical Sampling	2023	Clara Meister Tiago Pimentel Gian Wiher Ryan Cotterell

Works That Cite This (7)

Action	Title	Year	Authors
+ PDF Chat	Fidelity-Enriched Contrastive Search: Reconciling the Faithfulness-Diversity Trade-Off in Text Generation	2023	Weilin Chen Cheng-Kuang Wu Hsin‐Hsi Chen Chung-Chi Chen
+	Truncation Sampling as Language Model Desmoothing	2022	John K. Hewitt Christopher D. Manning Percy Liang
+	Lingxi: A Diversity-aware Chinese Modern Poetry Generation System	2023	Xinran Zhang Maosong Sun Jiafeng Liu Xiaobing Li
+	A Continuum of Generation Tasks for Investigating Length Bias and Degenerate Repetition	2022	Darcey Riley David Chiang
+	MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies	2023	Shiyue Zhang Shijie Wu Ozan İrsoy Steven Lu Mohit Bansal Mark Dredze David Rosenberg
+ PDF Chat	Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation	2023	Wen-Hong Zhu Hongkun Hao Rui Wang
+ PDF Chat	Fine-grained Conversational Decoding via Isotropic and Proximal Search	2023	Yuxuan Yao Han Wu Qiling Xu Linqi Song

Works Cited by This (15)

Action	Title	Year	Authors
+	A Simple, Fast Diverse Decoding Algorithm for Neural Generation	2016	Jiwei Li Will Monroe Dan Jurafsky
+ PDF Chat	Comparison of Diverse Decoding Methods from Conditional Language Models	2019	Daphne Ippolito Reno Kriz João Sedoc Maria Kustikova Chris Callison-Burch
+ PDF Chat	Hierarchical Neural Story Generation	2018	Angela Fan Mike Lewis Yann Dauphin
+ PDF Chat	Learning to Write with Cooperative Discriminators	2018	Ari Holtzman Jan Buys Maxwell Forbes Antoine Bosselut David Golub Yejin Choi
+ PDF Chat	Unifying Human and Statistical Evaluation for Natural Language Generation	2019	Tatsunori Hashimoto Hugh Zhang Percy Liang
+ PDF Chat	CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling	2019	Ning Miao Hao Zhou Lili Mou Rui Yan Lei Li
+	CTRL: A Conditional Transformer Language Model for Controllable Generation	2019	Nitish Shirish Keskar Bryan McCann Lav R. Varshney Caiming Xiong Richard Socher
+	Importance of Search and Evaluation Strategies in Neural Dialogue Modeling	2019	Ilia Kulikov Alexander Miller Kyunghyun Cho Jason Weston
+	The Curious Case of Neural Text Degeneration	2020	Ari Holtzman Jan Buys Leo Du Maxwell Forbes Yejin Choi
+	Plug and Play Language Models: A Simple Approach to Controlled Text Generation	2019	Sumanth Dathathri Andrea Madotto Janice Lan Jane Hung Eric Frank Piero Molino Jason Yosinski Rosanne Liu