Scalable problem localization for distributed systems: principles and practices

Problem localization is a critical part of providing crucial system management capabilities to modern distributed environments. One key open challenge is for problem localization solutions to scale for systems containing hundreds or even thousands of nodes, whilst still remaining fast enough to respond to rapid environment changes and sufficiently cost-effective to avoid overloading any management or application component. This paper meets the challenge by introducing two scalable frameworks applicable to a wide range of existing problem localization solutions: one based on a summary-driven, narrow-down procedure, the other through decomposing and decentralizing the problem localization process. Both frameworks, at their best, are able to achieve O(logN) problem localization time and O(1) per node communication load. The contrasting natures of both frameworks provide them with complimentary strengths that make them suitable for different scenarios in practice. We demonstrate our approaches in simulation settings and two real-world environments and show promising scalability benefits that can make a difference in system management operations.