Empirical Risk Minimization with f-Divergence Regularization in Statistical Learning

This report presents the solution to the empirical risk minimization with f -divergence regularization, under mild conditions on f .Under such conditions, the optimal measure is shown to be unique and to always exist.The solution is presented as a closed-form expression of the Radon-Nikodym derivative of the optimal probability measure with respect to the reference measure.Examples for particular choices of the function f are presented.For some choices, existing results are obtained as special cases of the main result.These include the unique solutions to the empirical risk minimization with relative entropy regularization (Type-I and Type-II).