(Nonparametric) Bayesian statistics

Bayesian analysis is a method of statistical inference that allows one to combine prior information about a population parameter with evidence from information contained in a sample to guide the statistical inference process. This is done by specifying a prior probability distribution for a parameter of interest and, thereafter, the evidence/data is obtained and combined through an application of Bayes’s theorem to provide a posterior probability distribution for this parameter. Bayesian nonparametrics, in particular, aims for efficient inferential procedures for infinite-dimensional models. Nowadays the probabilistic inference of massive and complex data has received much attention in statistics and machine learning, and Bayesian nonparametrics is one of their core tools.

Researchers

Dr.ir. G.N.J.C. (Joris) Bierkens

Prof.dr. A.J. (Annoesjka) Cabo

Dr. H.N. (Hanne) Kekkonen

Dr. J. (Jakob) Söhl

Prof.dr. A.W. (Aad) van der Vaart

Books

2017 - Subhashis Ghosal, Aad van der Vaart - Cambridge University Press

Fundamentals of Nonparametric Bayesian Inference

Explosive growth in computing power has made Bayesian methods for infinite-dimensional models - Bayesian nonparametrics - a nearly universal framework for inference, finding practical use in numerous subject areas. Written by leading researchers, this authoritative text draws on theoretical advances of the past twenty years to synthesize all aspects of Bayesian nonparametrics, from prior construction to computation and large sample behavior of posteriors. Because understanding the behavior of posteriors is critical to selecting priors that work, the large sample theory is developed systematically, illustrated by various examples of model and prior combinations. Precise sufficient conditions are given, with complete proofs, that ensure desirable posterior properties and behavior. Each chapter ends with historical notes and numerous exercises to deepen and consolidate the reader's understanding, making the book valuable for both graduate students and researchers in statistics and machine learning, as well as in application areas such as econometrics and biostatistics.

Key Publications

2022 - Hanne Kekkonen, Matti Lassas, Eero Saksman, Samuli Siltanen - Inverse Problems and Imaging

Random tree Besov priors – Towards fractal imaging

We propose alternatives to Bayesian prior distributions that are frequently used in the study of inverse problems. Our aim is to construct priors that have similar good edge-preserving properties as total variation or Mumford-Shah priors but correspond to well-defined infinite-dimensional random variables, and can be approximated by finite-dimensional random variables. We introduce a new wavelet-based model, where the non-zero coefficients are chosen in a systematic way so that prior draws have certain fractal behaviour. We show that realisations of this new prior take values in Besov spaces and have singularities only on a small set \(\tau\) with a certain Hausdorff dimension. We also introduce an efficient algorithm for calculating the MAP estimator, arising from the the new prior, in the denoising problem.

2021 - Joris Bierkens, Sebastiano Grazzi, Frank van der Meulen, Moritz Schauer - Statistics and Computing

A piecewise deterministic Monte Carlo method for diffusion bridges

We introduce the use of the Zig-Zag sampler to the problem of sampling conditional diffusion processes (diffusion bridges). The Zig-Zag sampler is a rejection-free sampling scheme based on a non-reversible continuous piecewise deterministic Markov process. Similar to the Lévy–Ciesielski construction of a Brownian motion, we expand the diffusion path in a truncated Faber–Schauder basis. The coefficients within the basis are sampled using a Zig-Zag sampler. A key innovation is the use of the fully local algorithm for the Zig-Zag sampler that allows to exploit the sparsity structure implied by the dependency graph of the coefficients and by the subsampling technique to reduce the complexity of the algorithm. We illustrate the performance of the proposed methods in a number of examples.

2019 - Richard Nickl, Jakob Söhl - Electronic Journal of Statistics

Bernstein–von Mises theorems for statistical inverse problems II: compound Poisson processes

We study nonparametric Bayesian statistical inference for the parameters governing a pure jump process of the form

\(Y_{t}=\sum_{k=1}^{N(t)}Z_{k},~~~t\ge 0,\)

where \(N(t)\) is a standard Poisson process of intensity \(\lambda\), and \(Z_k\) are drawn i.i.d. from jump measure \(\mu\). A high-dimensional wavelet series prior for the Lévy measure ν=λμ is devised and the posterior distribution arises from observing discrete samples \(Y_{\Delta},Y_{2\Delta},\dots,Y_{n\Delta}\) at fixed observation distance \(\Delta\), giving rise to a nonlinear inverse inference problem. We derive contraction rates in uniform norm for the posterior distribution around the true Lévy density that are optimal up to logarithmic factors over Hölder classes, as sample size n increases. We prove a functional Bernstein–von Mises theorem for the distribution functions of both \(\mu\) and \(\nu\), as well as for the intensity \(\lambda\), establishing the fact that the posterior distribution is approximated by an infinite-dimensional Gaussian measure whose covariance structure is shown to attain the information lower bound for this inverse problem. As a consequence posterior based inferences, such as nonparametric credible sets, are asymptotically valid and optimal from a frequentist point of view.

2017 - Richard Nickl, Jakob Söhl - The Annals of Statistics

Nonparametric Bayesian posterior contraction rates for discretely observed scalar diffusions

We consider nonparametric Bayesian inference in a reflected diffusion model \(dX_{t}=b(X_{t})\,dt+\sigma(X_{t})\,dW_{t}\), with discretely sampled observations \(X_{0},X_{\Delta},\ldots,X_{n\Delta}\). We analyse the nonlinear inverse problem corresponding to the “low frequency sampling” regime where \(\Delta>0\) is fixed and \(n\to\infty\). A general theorem is proved that gives conditions for prior distributions \(\pi\) on the diffusion coefficient \(\sigma\) and the drift function \(b\) that ensure minimax optimal contraction rates of the posterior distribution over Hölder–Sobolev smoothness classes. These conditions are verified for natural examples of nonparametric random wavelet series priors. For the proofs, we derive new concentration inequalities for empirical processes arising from discretely observed diffusions that are of independent interest.

2016 - Hanne Kekkonen, Matti Lassas, Samuli Siltanen - Inverse Problems

Posterior consistency and convergence rates for Bayesian inversion with hypoelliptic operators

The Bayesian approach to inverse problems is studied in the case where the forward map is a linear hypoelliptic pseudodifferential operator and measurement error is additive white Gaussian noise. The measurement model for an unknown Gaussian random variable \(U(x,\omega )\) is \({M}_{\delta }(y,\omega )=A\) \((U(x,\omega ))+\delta \phantom{\rule{.2mm}{0ex}}{ \mathcal E }(y,\omega )\), where A is a finitely many orders smoothing linear hypoelliptic operator and \(\delta \gt 0\) is the noise magnitude. The covariance operator CU of U is smoothing of order \(2r\), self-adjoint, injective and elliptic pseudodifferential operator. If \({ \mathcal E }\) was taking values in L2 then in Gaussian case solving the conditional mean (and maximum a posteriori) estimate is linked to solving the minimisation problem \({T}_{\delta }({m}_{\delta })={\mathrm{arg}\mathrm{min}}_{u\in {H}^{r}}\) \(\{\parallel {Au}-{m}_{\delta }{\parallel }_{{L}^{2}}^{2}+{\delta }^{2}\parallel {C}_{U}^{-1/2}u{\parallel }_{{L}^{2}}^{2}\}.\) However, Gaussian white noise does not take values in L2 but in \({H}^{-s}\) where \(s\gt 0\) is big enough. A modification of the above approach to solve the inverse problem is presented, covering the case of white Gaussian measurement noise. Furthermore, the convergence of the conditional mean estimate to the correct solution as \(\delta \to 0\) is proven in appropriate function spaces using microlocal analysis. Also the frequentist posterior contractions rates are studied.

2015 - Ismaël Castillo, Johannes Schmidt-Hieber, Aad van der Vaart - The Annals of Statistics

Bayesian linear regression with sparse priors

We study full Bayesian procedures for high-dimensional linear regression under sparsity constraints. The prior is a mixture of point masses at zero and continuous distributions. Under compatibility conditions on the design matrix, the posterior distribution is shown to contract at the optimal rate for recovery of the unknown sparse vector, and to give optimal prediction of the response vector. It is also shown to select the correct sparse model, or at least the coefficients that are significantly different from zero. The asymptotic shape of the posterior distribution is characterized and employed to the construction and study of credible sets for uncertainty quantification.

2008 - A. W. van der Vaart, J. H. van Zanten - The Annals of Statistics

Rates of contraction of posterior distributions based on Gaussian process priors

We derive rates of contraction of posterior distributions on nonparametric or semiparametric models based on Gaussian processes. The rate of contraction is shown to depend on the position of the true parameter relative to the reproducing kernel Hilbert space of the Gaussian process and the small ball probabilities of the Gaussian process. We determine these quantities for a range of examples of Gaussian priors and in several statistical settings. For instance, we consider the rate of contraction of the posterior distribution based on sampling from a smooth density model when the prior models the log density as a (fractionally integrated) Brownian motion. We also consider regression with Gaussian errors and smooth classification under a logistic or probit link function combined with various priors.

2002 - Subhashis Ghosal, Jayanta K. Ghosh, Aad W. van der Vaart - The Annals of Statistics

Convergence rates of posterior distributions

We consider the asymptotic behavior of posterior distributions and Bayes estimators for infinite-dimensional statistical models. We give general results on the rate of convergence of the posterior measure. These are applied to several examples, including priors on finite sieves, log-spline models, Dirichlet processes and interval censoring.