Research

2025

1. arXiv

   

   Conformal Inference for Open-Set and Imbalanced Classification

   Tianmin Xie , Yanfei Zhou, Ziyi Liang, Stefano Favaro, and Matteo Sesia

   In arXiv preprint, 2025

   📄 Paper Summary

   This paper presents a conformal prediction method for classification in highly imbalanced and open-set settings, where there are many possible classes and not all may be represented in the data. Existing approaches require a finite, known label space and typically involve random sample splitting, which works well when there is a sufficient number of observations from each class. Consequently, they have two limitations: (i) they fail to provide adequate coverage when encountering new labels at test time, and (ii) they may become overly conservative when predicting previously seen labels. To obtain valid prediction sets in the presence of unseen labels, we compute and integrate into our predictions a new family of conformal p-values that can test whether a new data point belongs to a previously unseen class. We study these p-values theoretically, establishing their optimality, and uncover an intriguing connection with the classical Good–Turing estimator for the probability of observing a new species. To make more efficient use of imbalanced data, we also develop a selective sample splitting algorithm that partitions training and calibration data based on label frequency, leading to more informative predictions. Despite breaking exchangeability, this allows maintaining finite-sample guarantees through suitable re-weighting. With both simulated and real data, we demonstrate our method leads to prediction sets with valid coverage even in challenging open-set scenarios with infinite numbers of possible labels, and produces more informative predictions under extreme class imbalance.

2024

1. NeurIPS

   

   Conformal Classification with Equalized Coverage for Adaptively Selected Groups

   Yanfei Zhou, and Matteo Sesia

   In Advances in Neural Information Processing Systems (NeurIPS), 2024

   📄 Paper Summary Code Poster

   This paper introduces a conformal inference method to evaluate uncertainty in classification by generating prediction sets with valid coverage conditional on adaptively chosen features. These features are carefully selected to reflect potential model limitations or biases. This can be useful to find a practical compromise between efficiency—by providing informative predictions—and algorithmic fairness—by ensuring equalized coverage for the most sensitive groups. We demonstrate the validity and effectiveness of this method on simulated and real data sets.

2. ICML

   

   Conformalized Adaptive Forecasting of Heterogeneous Trajectories

   Yanfei Zhou, Lars Lindemann, and Matteo Sesia

   In International Conference on Machine Learning (ICML), 2024

   📄 Paper Summary Code Poster

   We present a new conformal method for generating simultaneous forecasting bands guaranteed to cover the entire path of a new random trajectory with sufficiently high probability. Prompted by the need for dependable uncertainty estimates in motion planning applications where the behavior of diverse objects may be more or less unpredictable, we blend different techniques from online conformal prediction of single and multiple time series and ideas for addressing heteroscedasticity in regression. This solution is both principled, providing precise finite-sample guarantees, and effective, often leading to more informative predictions than prior methods.

2023

1. ICML

   

   Conformal inference is (almost) free for neural networks trained with early stopping

   Ziyi Liang , Yanfei Zhou, and Matteo Sesia

   In International Conference on Machine Learning (ICML), 2023

   📄 Paper Summary Code Poster

   We propose conformalized early stopping, a method that combines early stopping with conformal calibration while efficiently reusing a single hold-out set. This yields models that are accurate and provide reliable uncertainty quantification without multiple data splits. We develop practical implementations for outlier detection, multiclass classification, and regression.

2022

1. NeurIPS

   

   Training uncertainty-aware classifiers with conformalized deep learning

   Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia , and Yanfei Zhou

   In Advances in Neural Information Processing Systems (NeurIPS), 2022

   📄 Paper Summary Code Poster

   We develop a training strategy for deep multiclass classification that improves model calibration, leading to more informative, reliable uncertainty estimates. Its core is a novel loss function that mitigates overfitting and overconfidence often observed with cross-entropy training. We show the method’s reliability and efficiency on synthetic and real datasets such as CIFAR-10.