Understanding and Exploiting Network Traffic Redundancy

The Internet carries a vast amount and a wide range of content. Some
\nof this content is more popular, and accessed more frequently, than
\nothers. The popularity of content could be quite ephemeral - e.g., a
\nWeb flash crowd - or much more permanent - e.g., google.com's
\nbanner. A direct consequence of the skew in popularity is that, at any
\ntime, a fraction of the information carried over the Internet is
\nredundant.
\n
\nWe make two contributions in this paper. First, we study the
\nfundamental properties of the redundancy in the information carried
\nover the Internet, with a focus on network edges. We collect traffic
\ntraces at two network edge locations -- a large university's access
\nlink serving roughly 50,000 users, and a tier-1 ISP network link
\nconnected to a large data center. We conduct several analyses over
\nthis data: What fraction of bytes are redundant? What is the
\nfrequency at which strings of bytes repeat across different packets?
\nWhat is the overlap in the information accessed by distinct groups of
\nend-users? 
\n
\nSecond, we leverage our measurement observations in the design of a
\nfamily mechanisms for eliminating redundancy in network traffic and
\nimproving the overall network performance. The mechanisms we proposed
\ncan improve the available capacity of single network links as well as
\nbalance load across multiple network links.