A Neural Model of Adaptation in Reading

Marten van Schijndel, Tal Linzen

Type: Article

Publication Date: 2018-01-01

Citations: 49

DOI: https://doi.org/10.18653/v1/d18-1499

Abstract

It has been argued that humans rapidly adapt their lexical and syntactic expectations to match the statistics of the current linguistic context. We provide further support to this claim by showing that the addition of a simple adaptation mechanism to a neural language model improves our predictions of human reading times compared to a non-adaptive model. We analyze the performance of the model on controlled materials from psycholinguistic experiments and show that it adapts not only to lexical items but also to abstract syntactic structures.

Locations

arXiv (Cornell University) - View - PDF
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing - View - PDF

Similar Works

Action	Title	Year	Authors
+	A Neural Model of Adaptation in Reading	2018	Marten van Schijndel Tal Linzen
+	Context Limitations Make Neural Language Models More Human-Like	2022	Tatsuki Kuribayashi Yohei Oseki Ana Brassard Kentaro Inui
+ PDF Chat	Context Limitations Make Neural Language Models More Human-Like	2022	Tatsuki Kuribayashi Yohei Oseki Ana Brassard Kentaro Inui
+	On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior	2020	Ethan Wilcox Jon Gauthier Jennifer Hu Peng Qian Roger Lévy
+ PDF Chat	YanFarmerJaeger_EyeTrackingAdaptation	2019	SHAORONG YAN Thomas A. Farmer T. Florian Jaeger
+	Humans and language models diverge when predicting repeating text	2023	Aditya Vaidya Javier S. Turek Alexander G. Huth
+	Humans and language models diverge when predicting repeating text	2023	Aditya R. Vaidya Javier S. Turek Alexander G. Huth
+	Syntactic prediction adaptation accounts for language processing and language learning	2021	Naomi Havron Mireille Babineau Anne‐Caroline Fiévet Alex de Carvalho Anne Christophe
+	A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans	2021	Ethan Wilcox Pranali Vani Roger Lévy
+	A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans	2021	Ethan Wilcox Pranali Vani Roger Lévy
+	Prediction as a basis for skilled reading: insights from modern language models	2022	Benedetta Cevoli Chris Watkins Kathleen Rastle
+	Can training neural language models on a curriculum with developmentally plausible data improve alignment with human reading behavior?	2023	Aryaman Chobey Oliver Smith Anzi Wang Grusha Prasad
+	Can training neural language models on a curriculum with developmentally plausible data improve alignment with human reading behavior?	2023	Aryaman Chobey Oliver Smith Anzi Wang Grusha Prasad
+	Prediction as a Basis for Skilled Reading: Insights from Modern Language Models	2021	Benedetta Cevoli Chris Watkins Kathleen Rastle
+	Modeling Task Effects in Human Reading with Neural Network-based Attention	2018	Michael G. Hahn Frank Keller
+	Modeling Task Effects in Human Reading with Neural Network-based Attention	2022	Michael G. Hahn Frank Keller
+ PDF Chat	Is In-Context Learning a Type of Gradient-Based Learning? Evidence from the Inverse Frequency Effect in Structural Priming	2024	Zhenghao Zhou Robert Frank R. Thomas McCoy
+	Probing Language Models from A Human Behavioral Perspective	2023	Xintong Wang Xiaoyu Li Xingshan Li Chris Biemann
+	Surprisal does not explain syntactic disambiguation difficulty: evidence from a large-scale benchmark	2023	Kuan‐Jung Huang Suhas Arehalli Mari Kugemoto Christian Muxica Grusha Prasad Brian Dillon Tal Linzen
+ PDF Chat	Reverse-Engineering the Reader	2024	Samuel Kiegeland Ethan Wilcox Afra Amini David Robert Reich Ryan Cotterell

Works That Cite This (23)

Action	Title	Year	Authors
+	Language Model Evaluation Beyond Perplexity	2021	Clara Meister Ryan Cotterell
+	Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State	2019	Richard Futrell Ethan Wilcox Takashi Morita Peng Qian Miguel Ballesteros Roger Lévy
+	On the Opportunities and Risks of Foundation Models	2021	Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ B. Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill
+ PDF Chat	Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars	2022	Ryo Yoshida
+	On the Effect of Anticipation on Reading Times	2023	Tiago Pimentel Clara Meister Ethan G. Wilcox Roger Lévy Ryan Cotterell
+	UnNatural Language Inference	2021	Koustuv Sinha Prasanna Parthasarathi Joëlle Pineau Adina Williams
+ PDF Chat	Discourse Context Predictability Effects in Hindi Word Order	2022	Sidharth Ranjan Marten van Schijndel Sumeet Agarwal Rajakrishnan Rajkumar
+	Overestimation of Syntactic Representation in Neural Language Models	2020	Jordan Kodner Nitish Gupta
+	Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Models	2019	Grusha Prasad Marten van Schijndel Tal Linzen
+ PDF Chat	Comparing gated and simple recurrent neural network architectures as models of human sentence processing	2018	Christoph Aurnhammer Stefan L. Frank

Works Cited by This (10)

Action	Title	Year	Authors
+ PDF Chat	Expectation-based syntactic comprehension	2007	Roger Lévy
+ PDF Chat	Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies	2016	Tal Linzen Emmanuel Dupoux Yoav Goldberg
+	Dynamic Evaluation of Neural Sequence Models	2017	Ben Krause Emmanuel Kahembwe Iain Murray Steve Renals
+	Personalized Language Model for Query Auto-Completion	2018	Aaron Jaech Mari Ostendorf
+	Improving Neural Language Models with a Continuous Cache	2016	Édouard Grave Armand Joulin Nicolas Usunier
+	The Natural Stories Corpus	2017	Richard Futrell Edward Gibson Harry Tily Idan Blank Anastasia Vishnevetsky Steven T. Piantadosi Evelina Fedorenko
+	Colorless Green Recurrent Networks Dream Hierarchically	2018	Kristina Gulordava Piotr Bojanowski Édouard Grave Tal Linzen Marco Baroni
+	A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference	2018	Adina Williams Nikita Nangia Samuel Bowman
+	The Natural Stories Corpus	2017	Richard Futrell Edward Gibson Hal Tily Idan Blank Anastasia Vishnevetsky Steven T. Piantadosi Evelina Fedorenko
+	Pointer Sentinel Mixture Models	2016	Stephen Merity Caiming Xiong James T. Bradbury Richard Socher