Design of Hierarchical Neural Networks Using Deep LSTM and Self-Organizing Dynamical Fuzzy-Neural Network Architecture

Time series forecasting is an essential and challenging task, especially for large-scale time-series (LSTS) forecasting, which plays a crucial role in many real-world applications. Due to the instability of time series data and the randomness (noise) of their characteristics, it is difficult for polynomial neural network (PNN) and its modifications to achieve accurate and stable time series prediction. In this study, we propose a novel structure of hierarchical neural networks (HNN) realized by long short-term memory (LSTM), two classes of self-organizing dynamical fuzzy neural network architectures of fuzzy rule-based polynomial neurons (FPN) and polynomial neurons (PN) constructed by variant generation of nodes as well as layers of networks. The proposed HNN combines the deep learning method with the PNN method for the first time and extends it to time series prediction as a modification of PNN. LSTM extracts the temporal dependencies present in each time series and enables the model to learn its representation. FPNs are designed to capture the complex non-linear patterns present in the data space by utilizing Fuzzy C-Means clustering (FCM) and least square error (LSE)-based learning of polynomial functions. The self-organizing hierarchical network architecture generated by the Elitism-based Roulette Wheel Selection (ERWS) strategy ensures that candidate neurons exhibit sufficient fitting ability while enriching the diversity of heterogeneous neurons, addressing the issue of multicollinearity and providing opportunities to select better prediction neurons. In addition, L2-norm regularization is applied to mitigate the overfitting problem. Experiments are conducted on 9 real-world LSTS datasets including three practical applications. The results show that the proposed model exhibits high prediction performance, outperforming many state-of-the-art models.