Improving Scalability for Multi-Agent Reinforcement Learning via Attention Parameter Sharing

Scalability enhancement in Multi-agent Reinforcement Learning (MARL) is essential for tackling large-scale multi-agent challenges. Parameter sharing serves as an efficient mechanism to alleviate computational complexity during training, promoting improved learning efficiency and system stability. However, full parameter sharing often overlooks the differences between agents, leading to policy homogenization, slower convergence and difficulties in adapting to diverse decision-making tasks. In this paper, we propose a method called Attention Parameter Sharing (AtPS). AtPS incorporates multi-head self-attention into the value network, allowing agents to selectively focus on other agents with high similarity to themselves, which reduces model parameters while maintaining effective coordination. Moreover, we employ hierarchical clustering using the attention weights of each agent to form multi-agent groups, facilitating targeted parameter sharing within each group. Numerical results demonstrate that AtPS outperforms existing methods across various tasks in the six environments of SMAC, particularly in heterogeneous multi-agent scenarios. As the agent count grows, the performance gains of our method become increasingly evident, underscoring its effectiveness and practicality in scaling MARL.