ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

Petter Törnberg

Type: Preprint

Publication Date: 2023-01-01

Citations: 94

DOI: https://doi.org/10.48550/arxiv.2304.06588

View Publication

Locations

arXiv (Cornell University) - View - PDF
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks	2023	Fabrizio Gilardi Meysam Alizadeh Maël Kubli
+	ChatGPT outperforms crowd workers for text-annotation tasks	2023	Fabrizio Gilardi Meysam Alizadeh Maël Kubli
+	Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks	2023	Yiming Zhu Peixian Zhang Ehsan-Ul Haq Pan Hui Gareth Tyson
+	Chatbots Are Not Reliable Text Annotators	2023	Ross Deans Kristensen‐McLachlan Miceal Canavan Márton Kardos Mia Jacobsen Lene Aarøe
+ PDF Chat	Exploring the Capability of ChatGPT to Reproduce Human Labels for Social Computing Tasks (Extended Version)	2024	Yiming Zhu Peixian Zhang Ehsan-Ul Haq Pan Hui Gareth Tyson
+ PDF Chat	Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI	2024	Nicholas Pangakis Samuel Wolken
+ PDF Chat	Towards Hybrid Intelligence in Journalism: Findings and Lessons Learnt from a Collaborative Analysis of Greek Political Rhetoric by ChatGPT and Humans	2024	Thanasis Troboukis Kelly Kiki Antonis Galanopoulos Pavlos Sermpezis Stelios Karamanidis Ilias Dimitriadis Athena Vakali
+ PDF Chat	Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm	2025	Ljubiša Bojić Olga Zagovora Asta Zelenkauskaite Vuk Vuković Milan Čabarkapa Selma Veseljević Jerković Ana Jovančević
+ PDF Chat	Selecting Between BERT and GPT for Text Classification in Political Science Research	2024	Yu Wang Qu Wen Xin Ye
+ PDF Chat	Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing	2024	Mao Li Frederick G. Conrad
+	Towards Coding Social Science Datasets with Language Models	2023	Christopher Michael Rytting Taylor Sorensen Lisa P. Argyle Ethan C. Busby Nancy Fulda Joshua R. Gubler D. L. Wingate
+ PDF Chat	Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels	2024	Nicholas Pangakis Samuel Wolken
+ PDF Chat	PoliPrompt: A High-Performance Cost-Effective LLM-Based Text Classification Framework for Political Science	2024	Menglin Liu Ge Shi
+ PDF Chat	Can Large Language Models Transform Computational Social Science?	2023	Caleb Ziems William A. Held Omar Ahmed Shaikh Jiaao Chen Zhehao Zhang Diyi Yang
+ PDF Chat	Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Data	2024	Bastián González‐Bustamante
+ PDF Chat	Donald Trumps in the Virtual Polls: Simulating and Predicting Public Opinions in Surveys Using Large Language Models	2025	Chen Zhang Lijia Wei Song Jiang
+ PDF Chat	Donald Trumps in the Virtual Polls: Simulating and Predicting Public Opinions in Surveys Using Large Language Models	2024	Song Jiang Lijia Wei Chen Zhang
+	Can Large Language Models Transform Computational Social Science?	2023	Caleb Ziems William A. Held Omar Ahmed Shaikh Jiaao Chen Zhehao Zhang Diyi Yang
+ PDF Chat	Unmasking the Imposters: In-Domain Detection of Human vs. Machine-Generated Tweets	2024	Bryan Tuck Rakesh Verma
+ PDF Chat	Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways	2024	Shubham Atreja Joshua Ashkinaze Lingyao Li Julia Mendelsohn Libby Hemphill

Works That Cite This (20)

Action	Title	Year	Authors
+	A Large Language Model Approach to Educational Survey Feedback Analysis	2023	Michael Parker Caitlin Anderson Claire Stone YeaRim Oh
+ PDF Chat	Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models	2023	Junpeng Li Zixia Jia Zilong Zheng
+ PDF Chat	Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement	2023	Rosamond Thalken Edward H. Stiglitz David Mimno Matthew Wilkens
+	A survey of GPT-3 family large language models including ChatGPT and GPT-4	2023	Katikapalli Subramanyam Kalyan
+ PDF Chat	ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness	2023	Ján Čegiň Jakub Šimko Peter Brusilovsky
+ PDF Chat	AI for social science and social science of AI: A survey	2024	R. F. Xu Yingfei Sun Mengjie Ren Shiguang Guo Ruotong Pan Hongyu Lin Le Sun Xianpei Han
+ PDF Chat	Prompting a Large Language Model to Generate Diverse Motivational Messages: A Comparison with Human-Written Messages	2023	Samuel Rhys Cox Ashraf Abdul Wei Tsang Ooi
+	A Comparative Study on Annotation Quality of Crowdsourcing and LLm Via Label Aggregation	2024	Jiyi Li
+	GPT is an effective tool for multilingual psychological text analysis	2023	Steve Rathje Dan-Mircea Mirea Ilia Sucholutsky Raja Marjieh Claire Robertson Jay Joseph Van Bavel
+	Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design	2024	Lindia Tjuatja Valerie Chen Tongshuang Wu Ameet Talwalkwar Graham Neubig

Works Cited by This (0)

Action	Title	Year	Authors