Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, Lihua Lei

Type: Preprint

Publication Date: 2021-01-01

Citations: 9

DOI: https://doi.org/10.48550/arxiv.2110.01052

View Publication

Locations

arXiv (Cornell University) - View - PDF
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	Distribution-Free, Risk-Controlling Prediction Sets	2021	Stephen Bates Anastasios N. Angelopoulos Lihua Lei Jitendra Malik Michael I. Jordan
+ PDF Chat	Distribution-free, Risk-controlling Prediction Sets	2021	Stephen Bates Anastasios N. Angelopoulos Lihua Lei Jitendra Malik Michael I. Jordan
+	T-Cal: An optimal test for the calibration of predictive models	2022	Donghwan Lee Xinmeng Huang Hamed Hassani Edgar Dobriban
+ PDF Chat	Robust Conformal Outlier Detection under Contaminated Reference Data	2025	Meshi Bashari Matteo Sesia Yaniv Romano
+	Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers	2022	Ziyi Liang Matteo Sesia Wenguang Sun
+ PDF Chat	Data-driven Error Estimation: Upper Bounding Multiple Errors with No Technical Debt	2024	Sanath Kumar Krishnamurthy Susan Athey Emma Brunskill
+	Local Calibration: Metrics and Recalibration	2021	Rachel Luo Aadyot Bhatnagar Yu Bai Shengjia Zhao Huan Wang Caiming Xiong Silvio Savarese Stefano Ermon Edward Schmerling Marco Pavone
+ PDF Chat	A Confidence Interval for the $\ell_2$ Expected Calibration Error	2024	Yan Sun Pratik Chaudhari Ian Barnett Edgar Dobriban
+	Evaluating and Calibrating Uncertainty Prediction in Regression Tasks	2019	Dan Levi Liran Gispan Niv Giladi Ethan Fetaya
+	Top-label calibration and multiclass-to-binary reductions.	2021	Chirag Gupta Aaditya Ramdas
+ PDF Chat	Reassessing How to Compare and Improve the Calibration of Machine Learning Models	2024	Muthu Chidambaram Rong Ge
+	Calibrating Deep Neural Networks using Explicit Regularisation and Dynamic Data Pruning	2022	Ramya Hebbalaguppe Rishabh Patra Tirtharaj Dash Gautam Shroff Lovekesh Vig
+ PDF Chat	Evaluating and Calibrating Uncertainty Prediction in Regression Tasks	2022	Dan Levi Liran Gispan Niv Giladi Ethan Fetaya
+ PDF Chat	Calibrating Deep Neural Networks using Explicit Regularisation and Dynamic Data Pruning	2023	Rishabh Patra Ramya Hebbalaguppe Tirtharaj Dash Gautam R. Shroff Lovekesh Vig
+	Automatically Adaptive Conformal Risk Control	2024	Vincent Blot Anastasios N. Angelopoulos Michael I. Jordan Nicolas Brunel
+	Calibrated Selective Classification	2022	Adam Fisch Tommi Jaakkola Regina Barzilay
+ PDF Chat	Revisiting Confidence Estimation: Towards Reliable Failure Prediction	2024	Fei Zhu Xu-Yao Zhang Zhen Cheng Cheng‐Lin Liu
+	Calibration tests beyond classification	2021	David Widmann Fredrik Lindsten Dave Zachariah
+	Testing for Overfitting	2023	James Schmidt
+	Integrative conformal <i>p</i>-values for out-of-distribution testing with labelled outliers	2024	Ziyi Liang Matteo Sesia Wenguang Sun

Works That Cite This (8)

Action	Title	Year	Authors
+ PDF Chat	Principal Uncertainty Quantification with Spatial Correlation for Image Restoration Problems	2023	Omer Belhasin Yaniv Romano Daniel Z. Freedman Ehud Rivlin Michael Elad
+ PDF Chat	Quantum Conformal Prediction for Reliable Uncertainty Quantification in Quantum Machine Learning	2023	Sangwoo Park Osvaldo Simeone
+	Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling	2023	Léo Andéol Thomas Fel Florence De Grancey Luca Mossina
+ PDF Chat	Theoretical characterization of uncertainty in high-dimensional linear classification	2023	Lucas Clarté Bruno Loureiro Florent Krząkała Lenka Zdeborová
+ PDF Chat	Cross-Validation Conformal Risk Control	2024	Kfir M. Cohen Sangwoo Park Osvaldo Simeone Shlomo Shamai
+ PDF Chat	Conformal Autoregressive Generation: Beam Search with Coverage Guarantees	2024	Nicolas Deutschmann Marvin Alberts María Rodríguez Martínez
+	A gentle introduction to gradient-based optimization and variational inequalities for machine learning	2024	Neha S. Wadia Yatin Dandi Michael I. Jordan
+ PDF Chat	Conformal Loss-Controlling Prediction	2024	Di Wang Ping Wang Zhong Ji Xiaojun Yang Hongyue Li

Works Cited by This (52)

Action	Title	Year	Authors
+	Ratio Limit Theorems for Empirical Processes	2003	Evarist Giné Vladimir Koltchinskii Jon A. Wellner
+ PDF Chat	A conformal prediction approach to explore functional data	2013	Jing Lei Alessandro Rinaldo Larry Wasserman
+ PDF Chat	Non-Parametric Estimation II. Statistically Equivalent Blocks and Tolerance Regions--The Continuous Case	1947	John W. Tukey
+	Classification with confidence	2014	Jing Lei
+ PDF Chat	Determination of Sample Sizes for Setting Tolerance Limits	1941	S. S. Wilks
+	Statistical Prediction with Special Reference to the Problem of Tolerance Limits	1942	S. S. Wilks
+ PDF Chat	Cross-conformal predictors	2013	Vladimir Vovk
+ PDF Chat	An Extension of Wilks' Method for Setting Tolerance Limits	1943	Abraham Wald
+	On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities	2015	Vladimir Vapnik Alexey Chervonenkis
+	A graphical approach to sequentially rejective multiple test procedures	2008	Frank Bretz Willi Maurer Werner Brannath Martin Posch