Information-Theoretic Bounds on the Generalization Error and Privacy\n Leakage in Federated Learning

Machine learning algorithms operating on mobile networks can be characterized\ninto three different categories. First is the classical situation in which the\nend-user devices send their data to a central server where this data is used to\ntrain a model. Second is the distributed setting in which each device trains\nits own model and send its model parameters to a central server where these\nmodel parameters are aggregated to create one final model. Third is the\nfederated learning setting in which, at any given time $t$, a certain number of\nactive end users train with their own local data along with feedback provided\nby the central server and then send their newly estimated model parameters to\nthe central server. The server, then, aggregates these new parameters, updates\nits own model, and feeds the updated parameters back to all the end users,\ncontinuing this process until it converges.\n The main objective of this work is to provide an information-theoretic\nframework for all of the aforementioned learning paradigms. Moreover, using the\nprovided framework, we develop upper and lower bounds on the generalization\nerror together with bounds on the privacy leakage in the classical, distributed\nand federated learning settings.\n Keywords: Federated Learning, Distributed Learning, Machine Learning, Model\nAggregation.\n