COMS-E6998 Information Theory in Computer Science Columbia , Fall 2017

Administrativia Instructor : Omri Weinstein TA : Zhenrui Liao Lectures: Tuesday 4:10-6:20 pm (10-15 min break) . Office Hours: Omri: Tue, 2:30-3:30 pm (or by apt). Zhenrui: Wed 2pm-4pm . Recitations: Optional, possibly a couple to bring up to speed . Original motivation: Netflix, Amazon Instant Video (+pics?) Abstraction of typical scenario: clients miss information during transmissions of network . So this problem models a scenario where clients miss some informatino and NW wishes to minimize re-transmission rate, Course Level : 6K (graduate) -- proof heavy but not technical bootcamp. No prerequisites beyond mathematical maturity + probability theory. Books/HW/reading list – on website (http://infotheorytcs.wikischolars.columbia.edu) .

Grading HW: Bi-weekly (~5-6 in total) : 40% . Final Project : 40% . Scribing 1 lecture : 20% (at most 13 lectures, up to 2 students a lecture) . HW policy: - 5-6 exercises per HW, each totals to 125 pts. - Collaborations allowed as long as you write your own solution separately! (new concepts takes time to sink in, hard to keep up without practice). Original motivation: Netflix, Amazon Instant Video (+pics?) Abstraction of typical scenario: clients miss information during transmissions of network . So this problem models a scenario where clients miss some informatino and NW wishes to minimize re-transmission rate,

Final Research Projects Research/Reading based : Choose a course-related paper, summarize project and prepare 10-min talk (for final presentations day). You’re expected to say “something new” about problem (connection/extension). “Meta project”: related to storage, compression and search in modern file systems (more on this soon). Project based on unexplored problem in dynamic data structures – both theory and applied aspects (many directions to choose from, more on this soon..) Implementation based (storage) : Focus on 1 sub-topic in storage project, implement algorithm on real data repositories (more soon). Small groups allowed. Original motivation: Netflix, Amazon Instant Video (+pics?) Abstraction of typical scenario: clients miss information during transmissions of network . So this problem models a scenario where clients miss some informatino and NW wishes to minimize re-transmission rate, - Consult + converge early on (e.g., during office hours or schedule meeting) - Project proposal deadline : October 10th.

Introduction

Motivating story for this course Engineering perspective: Traditional “home” of Information Theory (IT). IT underlies most modern digital technology (e.g., compression, coding, distributed storage…). Explore some of this theory, applications, extensions. Complexity perspective: Computational Complexity: “How many resources required to solve given computational problem?” (space, time, energy, neural layers, etc.) Holy grail : Unconditional lower bounds on such resources. Unfortunately, majority of Complexity built on conditional hardness results  Original motivation: Netflix, Amazon Instant Video (+pics?) Abstraction of typical scenario: clients miss information during transmissions of network . So this problem models a scenario where clients miss some informatino and NW wishes to minimize re-transmission rate, One of major achievements of TCS : understanding “information bottleneck” of computational models (as opposed to the efficiency of processing data).

The Complexity Theoretic View Course “birthright” : increasing role of information-theoretic mindset and methods in understanding computation (hopefully twist your minds as well). Most topics we’ll encounter are active research fields (e.g., streaming, LPs, circuits complexity, static/dynamic DS...). Many open problems… Original motivation: Netflix, Amazon Instant Video (+pics?) Abstraction of typical scenario: clients miss information during transmissions of network . So this problem models a scenario where clients miss some informatino and NW wishes to minimize re-transmission rate, What are we about to see in this course ? …

Data Compression and Interactive Compression Data compression, a.k.a Shannon’s “source coding” problem (today) : What is the essential amount of communication communication required to transmit/encode a random dataset (mention few natural and practical compression schemes, e.g., Huffman, JPEG). Later in the course: Does Shannon’s compression theory extend to interactive scenarios, i.e., “can we compress conversations”?

Streaming Real-time systems (financial databases, monitoring networks) store huge, rapidly changing data, which is too large to store explicitly. x1 x2 xn x +3 +17 +3 - 6 -15 +3 Long sequence of updates in [M] (e.g., IP addresses of packets passing through a router, DNA sequence, transactions in sensor networks…) Goal: Compute some function f(x) of the final stream (whp), with minimum space. Trivial solution: ~ n lg M = O(n) bits of space – too expensive  Relaxation: approximate f(x) (whp) using sublinear space. (ex: Fp := i |xi|p)

Data Structures : Time-Space tradeoffs Nearest Neighbor problem (cornerstone in most ML apps, optimization, DBs…) Rd Preprocess n pts into smallest possible memory, allowing fast answering of exp(d) >> n queries. ? Trivial solutions: Either exp(d) space (store all answers), or ~ n time (scan entire database) . Original motivation: Netflix, Amazon Instant Video (+pics?) Abstraction of typical scenario: clients miss information during transmissions of network . So this problem models a scenario where clients miss some informatino and NW wishes to minimize re-transmission rate, Are there better tradeoffs ? Some databases are dynamic – maintain them cheaply, online (like streaming)…

Compression and Storage “Dictionary” problem: Dropbox server needs to store file collection X = (X1, X2, …, Xn) » ¹ of multiple users, s.t individual files can be retrieved quickly (few memory accesses) . This field will bring us to a specific natural DS problem that arises in Storage systems. Realistic scenario – Files correlated (Info¹(X) ¿ i Info¹ (Xi) is space benchmark) . Tradeoffs b/w storage space and decoding time (# memory accesses) “locally decodable” compression schemes??

We’ll see later in the course: - Theoretical problem is mostly open (understanding it in general notoriously hard) - For some joint distributions, hopeless to have small space and fast (“local”)decoding… But, wait a minute – real life distributions aren’t like that. 2 natural and important data types (distributions) : (1) Visual ; (2) Genomic . (correlations are huge and dictionaries are important) Final project proposal: Theoretical and Applied. Suggested projects : (a) Define generative model for correlated file albums per data type, then try to design similarity search and compression algs (ideally dynamic). Work with real data repositories. (b) how to search ? (c) how to compress? (d) realistic problem is dynamic – can we handle updates? Original motivation: Netflix, Amazon Instant Video (+pics?) Abstraction of typical scenario: clients miss information during transmissions of network . So this problem models a scenario where clients miss some informatino and NW wishes to minimize re-transmission rate, Will host special session (40-min) on this project late September (doodle poll).

Interactive Compression Data compression (today’s topic). Does Shannon’s classical extend to interactive scenarios? Can we compress conversations ? Linear-Program LBs LPs = powerful algorithmic tool for solving (approximating) combinatorial optimization problems. How large (# vars/constraints) does an LP need to be in order to produce a good sln to resource allocation/packing problems? Original motivation: Netflix, Amazon Instant Video (+pics?) Abstraction of typical scenario: clients miss information during transmissions of network . So this problem models a scenario where clients miss some informatino and NW wishes to minimize re-transmission rate, Common denominator of all these applications? Communication and Information Theory (surprisingly, all implicitly embed some distributed problem over a channel…)

Rough plan (syllabus) Information Theory 101, Data Compression. Communication Complexity. Information Complexity and interactive compression. Applications to Streaming. Applications to Data Structures (static & dynamic). Original motivation: Netflix, Amazon Instant Video (+pics?) Abstraction of typical scenario: clients miss information during transmissions of network . So this problem models a scenario where clients miss some informatino and NW wishes to minimize re-transmission rate,