Speaker: Parikshit Gopalan 

Title:  Calibration through the lens of indistinguishability.

Abstract: Calibration is a classical notion from the forecasting literature which aims to address the question: how should predicted probabilities be interpreted? In a world where we only get to observe (discrete) outcomes, how should we evaluate a predictor that hypothesizes (continuous) probabilities over possible outcomes? The study of calibration has seen a surge of recent interest, driven by its importance in ML and applications to questions in theory. In this talk, we will explore some foundational questions about how we should define and measure calibration error, and some applications to learning and complexity theory.

Speaker: Nikolaos Ignatiadis 

Title: Empirical partially Bayes multiple testing and compound χ² decisions

Abstract: A common task in high-throughput biology is to screen for associations across thousands of units of interest, e.g., genes or proteins. Often, the data for each unit are modeled as Gaussian measurements with unknown mean and variance and are summarized as per-unit sample averages and sample variances. The downstream goal is multiple testing for the means. In this domain, it is routine to “moderate” (that is, to shrink) the sample variances through parametric empirical Bayes methods before computing p-values for the means. Such an approach is asymmetric in that a prior is posited and estimated for the nuisance parameters (variances) but not the primary parameters (means). Our work initiates the formal study of this paradigm, which we term “empirical partially Bayes multiple testing.” In this framework, if the prior for the variances were known, one could proceed by computing p-values conditional on the sample variances—a strategy called partially Bayes inference by Sir David Cox. We show that these conditional p-values satisfy an Eddington/Tweedie-type formula and are approximated at nearly-parametric rates when the prior is estimated by nonparametric maximum likelihood. The estimated p-values can be used with the Benjamini-Hochberg procedure to guarantee asymptotic control of the false discovery rate. Even in the compound setting, wherein the variances are fixed, the approach retains asymptotic type-I error guarantees.

Speaker: Licong Lin

Title: Breaking the quadratic barrier for $Z$-estimators via the jackknife.

Abstract: Resampling methods are especially well-suited to inference with estimators that provide only “black-box” access. Jackknife is a form of resampling, widely used for bias correction and variance estimation, that is well-understood under classical scaling where the sample size $n$ grows for a fixed problem. We study its behavior in application to estimating functionals using high-dimensional $Z$-estimators, allowing both the sample size $n$ and problem dimension $d$ to diverge. We begin showing that the plug-in estimator based on the $Z$-estimate suffers from a quadratic breakdown: while it is $\sqrt{n}$-consistent and asymptotically normal whenever $n\gtrsim d^2$, it fails for a broad class of problems whenever $n \lesssim d^2$. We then show that under suitable regularity conditions, applying a jackknife correction yields an estimate that is $\sqrt{n}$-consistent and asymptotically normal whenever $n\gtrsim d^{3/2}$. This provides strong motivation for the use of jackknife in high-dimensional problems where the dimension is moderate relative to sample size. We illustrate consequences of our general theory for various specific $Z$-estimators, including non-linear functionals in linear models; generalized linear models; and the inverse propensity score weighting (IPW) estimate for the average treatment effect, among others.

Speaker: Rohan Hore

Title: Conformal changepoint localization

Abstract: We study the problem of offline changepoint localization, where the goal is to identify the index at which the data-generating distribution changes. Existing methods often rely on restrictive parametric assumptions or asymptotic approximations, limiting their practical applicability. To address this, we propose a distribution-free framework, CONformal CHangepoint localization (CONCH), which leverages conformal p-values to efficiently construct valid confidence sets for the changepoint. Under mild assumptions of exchangeability within each segment and independence across segments, CONCH guarantees finite-sample coverage. We further derive principled score functions that yield informative confidence sets. With appropriate score functions, we prove that the normalized length of the confidence set indeed shrinks to zero. We further establish a universality result showing that any distribution-free changepoint localization method can be viewed as an instance of CONCH. Experiments on synthetic and real data confirm that CONCH delivers precise and reliable confidence sets even in challenging settings.

Speaker: Hamed Hassani

Title: Uncertainty Quantification for Generative and Collaborative AI

Abstract: As AI models become increasingly powerful, their predictions are now being integrated into real-world decision-making pipelines — from medicine to autonomous systems. Ensuring that these systems remain reliable requires a principled understanding of uncertainty, especially as modern AI moves toward generative models and collaborative settings where humans interact with AI. Together, these trends raise fundamental questions about how uncertainty should be quantified when decisions are made jointly by humans and machines, and how such quantification can extend to the complex output spaces of generative AI. This talk will focus on conformal prediction as a tool for uncertainty quantification and will aim to provide answers to these questions.

Speaker: Yifan Wu 

Title: Making and Evaluating Calibrated Forecasts

Abstract: Calibrated predictions can be reliably interpreted as probabilities. An important step towards achieving better calibration is to design an appropriate calibration measure to meaningfully assess the miscalibration level of a predictor. A recent line of work initiated by Haghtalab et al. [2024] studies the design of truthful calibration measures: a truthful measure is minimized when a predictor outputs the true probabilities, whereas a non-truthful measure incentivizes the predictor to lie so as to appear more calibrated. In this talk, I will present a perfectly truthful calibration measure in the batch setting, introduced in Hartline et al. [2025].

The perfectly truthful calibration measure can be generalized to multi-class prediction tasks. We study common methods of extending calibration measures from binary to multi-class prediction and identify ones that do or do not preserve truthfulness. In addition to truthfulness, we mathematically prove and empirically verify that our calibration measure exhibits superior robustness: it robustly preserves the ordering between dominant and dominated predictors, regardless of the choice of hyperparameters (bin sizes). This result addresses the non-robustness issue of binned ECE, which has been observed repeatedly in prior work.

Speaker: Anirban Chatterjee 

Title: Detecting Miscalibration: A Nearest Neighbor Approach

Abstract: In this talk, we introduce a nearest-neighbor based approach for testing the calibration of predictive models. For classification models, we focus on the well-known $L_2$ expected calibration measure and propose a resampling-based, consistent estimator of this measure using a nearest neighbor framework. The proposed estimator is consistent for the true calibration measure under minimal assumptions. Furthermore, we show that under the null hypothesis of a calibrated model, the properly normalized statistic is asymptotically standard Gaussian. This result enables a practical and readily applicable test for model calibration. In addition, we present a method to derandomize the resampling procedure, which can potentially improve the power of the test. Finally, we discuss extensions of this measure for assessing calibration beyond classification models.