Microsoft Technology Reduces Network Redundancy

11/24/2009 4:18:09 PM

Mon, November 23, 2009 — IDG News Service — Researchers at Microsoft Research India have developed a compression and redundancy elimination technology that can operate as a host service in enterprise systems without the use of accelerator devices over a WAN.

The project is called Coconet for Content Compression in Networks. The researchers monitored access links at 11 corporate enterprise locations for several days, as well as the access link of the University of Wisconsin, which had some of its students collaborate on the project.

Microsoft researchers estimated that about 75 percent of the bandwidth saved using redundancy elimination devices on a WAN comes from removing redundant byte-strings from within each client's traffic, Ramachandran Ramjee, project leader of Coconet, said in a telephone interview on Monday.

This kind of redundancy, described by the Microsoft researchers as intra-user redundancy, consists, for example, of the same user getting different versions of the same files from a server, or going to the same Web sites repeatedly to get an update, Ramjee said. This pattern presented the opportunity to move the redundancy elimination function to software running on end hosts on the network, reducing the need for deployment of expensive accelerator devices, or other redundancy elimination "middle-boxes" on the WAN, according to Ramjee.

Many large companies, having branch offices across the world, are consolidating their IT resources in a few locations to save on administrative costs, Ramjee said. "Instead of putting servers in each branch office they are consolidating in a few data centers in a few locations," he said.

A result of this move is that what used to be traffic on the LAN from a user's PC to a server in the branch office has now become relatively expensive WAN traffic, according to Ramjee. The move for consolidation has increased demand for products like WAN accelerator devices, which among other things also reduce redundancy in network traffic, he added.

The software developed by Microsoft Research India has been designed to be asymmetric so that the processing is done at the server, Ramjee said. It does not conflict with encryption on the network, because the removal of a redundancy at the server end is done before encryption, he added.

The software runs on general purpose servers and requires a typical cache size of about 10MB on the server for each client, and another 10MB at the client end. This ensures that the technology can also be used when the client is a resource-constrained device like a smartphone, Ramjee said.

The host service can identify and suppress redundancy as small as 32 bytes in the packets, according to Ramjee. If any bytes are found to match between a data request and previous data transmitted, the server indicates that the data is already on the client and redirects to it, he said.