What is the Value of Heuristic Model Selection and Aggregation in Federated Learning?

Federated learning (FL) is a promising distributed machine learning framework for mobile networks, where an aggregation server produces a global model by aggregating the local models from clients over multiple rounds. Usually, selecting a subset of clients instead of all improves communication efficiency. Therefore, several heuristic strategies for model selection and aggregation have been proposed to improve the global model accuracy or speed up model convergence. However, the effectiveness of these strategies lacks theoretical backing. To investigate this, an empirical comparison study that systematically and quantitatively evaluates existing heuristic strategies is conducted in this paper. Specifically, a FL prototype including nine model selection and aggregation strategies is developed. Experiments with three levels of non-IID data settings on this prototype reveal a trade-off between convergence stability and global model accuracy. Notably, selecting local models with max parameter entropy achieves an excellent balance between model accuracy and convergence stability when handling non-IID data. These findings contribute to a better understanding of heuristic model selection and aggregation strategies, offering valuable guidance for future FL development.