How we evaluate AI systems, and what the verdicts are worth.
Every claim about an AI system rests on a measurement. My research asks how much of a verdict comes from the system and how much from the protocol used to test it, and builds the artifacts that make the answer checkable: reporting standards for fairness audits of generative models, a real-world benchmark for fairness in online advertising, and offline estimators that judge a treatment policy before it is deployed. Evaluation is also the interface between research and regulation, since a rule that cannot be measured cannot be enforced. Earlier work on the statistical foundations of deep networks sits underneath all of it.
Generative models are routinely audited for bias, yet the verdicts rarely agree across papers or translate into deployment decisions. Prompt family, decoding temperature, refusal handling, and scoring rubric each shift the verdict substantially, and current reporting leaves these degrees of freedom implicit.
We quantify how much of a fairness conclusion comes from evaluation choices rather than from the model, diagnose the recurring failure modes (prompt sensitivity, sampling instability, counterfactual inconsistency, intersectional blind spots), and propose Fairness Cards as a standardized reporting artifact. A Fairness Card pins down the audit surface so that two audits of the same model can be compared.
In a controlled audit of Qwen2.5-7B-Instruct across 3,200 generations on a factorial grid of demographic slices, prompt families, decoding regimes, and seeds, the same model received opposite fairness verdicts depending solely on which prompt family was used to evaluate it. Ongoing work repeats the audit across several model families to measure how unstable such verdicts are in general, and extends the protocol to image generation.
On the mitigation side, Debias Anything guides a frozen diffusion model towards target attribute proportions while keeping its outputs diverse, using a pretrained vision-language embedding space in place of sensitive-attribute annotations.
Fairness in machine learning is tested most often on a handful of small academic tables. The systems people interact with daily (job recommendation, advertising auctions, content feeds) run on industrial-scale tabular data where the sensitive attribute is unavailable at training or inference time, so the usual fairness metrics cannot even be computed without a proxy.
FairJob, published in the NeurIPS Datasets and Benchmarks track, is a real-world benchmark for fairness in job advertising built from production logs, with a method for deriving protected-attribute proxies under privacy constraints and an evaluation of what bias mitigation costs in utility. On the mitigation side, we relate statistical independence measures (Cramér-von Mises, HSIC, distance correlation) to standard fairness criteria such as demographic parity and equalized odds, and turn these connections into differentiable regularizers that give practitioners direct control over the fairness-utility trade-off.
Personalization at industrial scale is a causal problem: what is the effect of an action (a bid, a recommendation, a discount) on an individual, and how should actions be allocated under a budget? The evaluation question is just as hard as the learning question. A new policy can only be judged on logged data collected under an old one, so every claim about its value is a counterfactual estimate.
CUVET exploits the diminishing-returns relationship between spend and value in advertising to derive a closed-form optimal policy via recursive partitioning that equalizes the marginal cost per unit value across user cohorts, and evaluates it against strong and naive baselines on a public policy dataset. For discrete allocation, we study how to maximize the success probability of a policy allocation, which gives operators a way to reason about risk rather than expected value alone.
What distribution does a deep network implicitly place on its outputs before seeing any data? With Gaussian priors on weights, the output prior becomes heavier-tailed as depth increases, following a generalized Weibull-tail pattern. The result matters for prior design and for calibration, which is itself an evaluation question: how far a model's stated uncertainty can be trusted.
During a PhD at Inria Grenoble, we established the connection between BNN priors and Weibull-tail distributions, introduced the Sub-Weibull class of distributions as a unifying framework generalizing sub-Gaussian and sub-exponential properties, and characterized how units in deep networks become dependent as depth grows. This line of work is collected and extended in a review of Bayesian neural networks published in Statistical Science.
Three threads shape the current work. The first asks how stable a fairness verdict is: we re-run the same audit across model families, releases, and benchmarks to separate what the model contributes from what the protocol contributes, and extend the protocol from text to diffusion models, where standard evaluation choices fail in the same way.
The second thread is behavioural evaluation of language models. We measure how a model's answers shift when other participants in a conversation assert a different answer, and whether those shifts replicate across model releases and benchmarks. Small changes in the stimulus or the scoring rule move the estimate, and a large part of the work is designing protocols where the measurement itself does not decide the result.
The third thread follows models into agentic settings. Once a model plans, calls tools, and acts over several steps, the space of behaviours grows combinatorially and the gap between offline benchmarks and deployed behaviour widens. How to audit, constrain, and certify such systems, with methods that regulators can point to, is the open problem I find most pressing. The longer-term goal is evaluation as a research area in its own right, with its own methods, failure taxonomies, and reproducibility standards, grounded in industry practice and feeding into the rules now being written for AI.