Finite Mixture Approximation with Applications to Density Estimation and Generalised ReLU Networks

Abstract

Finite mixture models are useful devices for approximating probability density functions using convex combinations of translated and rescaled density kernels. We first present qualitative results showing that location–scale finite mixtures can approximate broad classes of probability density functions in uniform and Lebesgue-space norms. We then use a smoothing and quantisation argument to derive explicit approximation rates over Sobolev classes and to obtain fast least-squares density estimators. We also discuss approximation by finite Gaussian mixtures in Kullback–Leibler divergence, where the existence of a finite second moment arises as a universal necessary condition. Finally, we show that the same smoothing and quantisation mechanism can be used to construct generalised ReLU networks for whole-space approximation and nonparametric regression on unbounded domains. In particular, under suitable regularity conditions, the resulting regression estimators attain the classical Sobolev minimax exponent up to a logarithmic factor.

Biography

Hien Nguyen is an Associate Professor in Applied Statistics at La Trobe University, a Professor at the Institute of Mathematics for Industry at Kyushu University, and Deputy Director of La Trobe University’s Centre for Technology Infusion. He undertakes editorial work for several journals, including the Australian & New Zealand Journal of Statistics, Statistical Analysis and Data Mining, and Heliyon. His research interests include approximation theory, nonparametric and computational statistics, mixture models, statistical learning, and optimisation.