The datacenter needs an operating system

Clusters of commodity servers have become a major computing platform, powering not only some of today’s most popular consumer applications—Internet services such as search and social networks—but also a growing number of scientific and enterprise workloads [2]. This rise in cluster computing has even led some to declare that “the datacenter is the new computer” [16, 24]. However, the tools for managing and programming this new computer are still immature. This paper argues that, due to the growing diversity of cluster applications and users, the datacenter increasingly needs an operating system.1 We take a broad view of an operating system as both a software layer that manages and abstracts hardware and a package of tools, such as programming languages and debuggers, that facilitate the use of a computer. Traditional OSes for a single computer play several key roles. First, they enable resource sharing between programs and users, allowing a computer to be used for multiple tasks and to respond interactively to several users. Second, they enable data sharing between programs, through abstractions such as pipes and files, so that users can combine independently written applications to solve problems. Third, they provide programming abstractions that simplify software development. Finally, OSes include system-wide debugging and monitoring facilities. Together, these features have allowed singlecomputer operating systems to foster a rich ecosystem of interoperable applications that we now take for granted. Much like the first computers, the first datacenters ran a small number of applications written by expert users. This situation is changing now, and we believe that datacenters need an OS-like layer for the same reason computers did: a rising diversity of applications and users. On the application side, there is an increasing number of cluster computing frameworks (e.g., MapReduce [10], Dryad [17], Pregel [21]) and storage systems [12, 9, 14], each suited for different use cases. Ideally, an organization should be able to dynamically share resources between these applications, and to easily exchange data between them (e.g., have a job that combines MapReduce