Conformal prediction

In the remaining part of the course, we shall focus on models’ predictions with uncertainties. More precisely, given a predictive model and a new data point, our goal is to construct an interval that has a high probability of containing the outcome associated with the data point. To do this, we will use conformal prediction, a frequentist method of constructing a prediction interval that relies on minimal assumption on the data distribution. In a sense, conformal prediction is an exact opposite of predictive Bayesian inference, which heavily relies on distributional assumptions of the model’s parameters (through the prior) and that of the data (through the likelihood). Of course, if the data distribution exactly matched our assumptions, the posterior predictive distribution would give us an accurate prediction interval. On the other hand, if our Bayesian model was misspecified, then the conformal prediction would give us a better prediction interval.

In the following chapters, we will cover fundamental ideas of conformal prediction in the context of regression and classification. We will discuss its computational issue and, introduce a couple of methods that are designed to solve this issue.