Information Theory in an Industrial Research Lab Marcelo J. Weinberger Information Theory Research Group Hewlett-Packard Laboratories – Advanced Studies Palo Alto, California, USA with contributions from the ITR group Purdue University – November 19, 2007

Information Theory research in the industry  Mission Research the mathematical foundations and practical applications of information theory, generating intellectual property and technology for “XXX Company” through the advancement of scientific knowledge in these areas  Apply the theory and work on the applications makes obvious sense for “XXX Company” research labs; But why invest on advancing the theory? some simple answers which apply to any basic research area: long-term investment, prestige, visibility, give back to society... this talk will be about a different type of answer: differentiating technology vs. enabling technology Main claim: working on the theory helps developing analytical tools that are needed to envision innovative, technology-differentiating ideas

Case studies  JPEG-LS: from universal context modeling to a lossless image compression standard  DUDE (Discrete Universal DEnoiser): from a formal setting for universal denoising to actual image denoising algorithms  Error-correcting codes in nanotechnology: the advantages of interdisciplinary research  2-D information theory: looking into the future of storage devices compress store, transmit de- compress Input Output

 The goal: upon observing, choose to optimize some fidelity criterion (e.g.: minimize number of symbol errors, squared distance, etc.)  A natural extension of work on prediction/compression  Applications: image and video denoising, text correction, financial data denoising, DNA sequence analysis, wireless communications… discrete source discrete memoryless channel (noise) denoiser Discrete Universal DEnoising (DUDE)

DUDE: how it’s done  pass 1: - gather statistics on symbol occurrences per context pattern - estimate noiseless symbol distribution given context pattern and noisy sample (posterior distribution)  pass 2: denoise each symbol, based on estimated posterior who do you believe? what you see, or what the global stats tell you? precise decision formula proven asymptotically optimal context template size must be carefully chosen zizi “context” samples sample being denoised data sequence noisy channel

 Key component of DUDE: Model the conditional distribution P ( Z i |context of Z i ) and infer P ( X i | Z i and context of Z i ) from it  Main issue: large alphabet large number of model parameters high learning cost  Leveraged “semi-universal’’ approach from image compression: rely on prior knowledge. Main tools: prediction contexts based on quantized data parameterized distributions  State-of-the-art for “salt-and-pepper” noise removal Competitive for Gaussian and ``real world” noise removal, but still room for improvement The main challenge in image denoising

Application 1: Image denoising  Best previous result in the literature: PSNR = 35.6 error rate=30% (Chan,Ho&Nikolova, IEEE IP Oct’05) error rate=30% PSNR=10.7 dB (``salt and pepper” noise) dude-denoised PSNR=38.3 dB

Application 2: Denoiser-enhanced ECC  Suitable for wireless communications  Leaves overall system ``as-is’’, but enhances receiver by denoising signal prior to error correction (ECC) decoding  Allows to design a “better receiver” that will recover signals other receivers would reject as undecodable transmitted codeword decodable region for regular ECC (handles code redundancy, structured) received noisy codeword denoising (handles source redundancy, natural) non-enhanced (no reception) DUDE-enhanced