Actor-Critic Methods for IRS Design in Correlated Channel Environments: A Closer Look Into the Neural Tangent Kernel of the Critic

The paper studies the design of an Intelligent Reflecting Surface (IRS) in order to support a Multiple-Input-Single-Output (MISO) communication system operating in a mobile, spatiotemporally correlated channel environment. The design objective is to maximize the expected sum of Signal-to-Noise Ratio (SNR) at the receiver over an infinite time horizon. The problem formulation gives rise to a Markov Decision Process (MDP). We propose an actor-critic algorithm for continuous control that accounts for both channel correlations and destination motion by constructing the state of the Reinforcement Learning algorithm to include history of destination positions and IRS phases. To account for the variability of the underlying value function, arising due to the channel variability, we propose to pre-process the input of the critic with a Fourier kernel, which enables stability in the process of neural value approximation. We also examine the use of the destination SNR as a component of the designed MDP state, which constitutes common practice in previous works. We empirically show that, when the channels are spatiotemporally varying, including the SNR in the state representation causes divergence. We provide insight on the aforementioned divergence by demonstrating the effect of the SNR inclusion on the Neural Tangent Kernel of the critic network. Based on our study, we propose a framework for designing actor-critic methods for IRS design and also for more general problems, that is predicated upon sufficient conditions of the critic’s Neural Tangent Kernel for convergence under neural value learning.