Shared-Dictionary Compression over HTTP (SDCH)‏ Wei-Hsin Lee June 2008

Why do we care? Speeding up Google and the Web –The faster the Web is, the more useful it is. –The faster Google web search is, the more searches people do. –Lots of users still suffer from slow networks. For example, in developing countries.

Reduce transmission time Reducing payload size is the key. Gzip works well as the compression for each individual response. What about common data shared by a group of pages (inter-response redundancy) or pages that change a little bit frequently? Only transmit the data that is common to each response once. Thereafter, send only the parts of the response that differ.

Why not RFC 3229? RFC3229 “Delta Compression in HTTP” –Good for saving bandwidth But –Too many states for server to track The possible states of is bigger than all possible search results. –Only applicable to the same URL Discourages aggressive caching. –No benefit for similar pages that don’t share an URL.

Shared-Dictionary Compression over HTTP (SDCH)‏ An addition to HTTP Small set of states (dictionaries) shared between client and server. Dictionaries are scoped by domain name and path. Just like cookies. It allows dictionaries to apply to multiple URLs.

SDCH protocol details SDCH defines –How client informs server of its capability and state. –How the server should respond to client when the client is SDCH capable. –How dictionaries get loaded into client. Implement VCDIFF (RFC 3284) differential compression format with enhancements –Interleave instructions with data so that each network packet can be decoded as it arrives. (chunked encoding) –Checksum to ensure data integrity

Example 1

Example 2

Other details Complement to Gzip or Deflate. –Should be used before applying Gzip Lab result –About 40 percent data reduction better than Gzip alone on Google search. –See faster Google search results. Especially under low bandwidth and high latency condition. Working on the best way to get this out to users.

Your help counts! Please join the group – –Protocol spec, and the encoder/decoder code will be there soon. Getting your hands dirty is even better! –Make your web site use SDCH. –Make Squid or Apache web servers SDCH capable.

Don’t forget to join the group.