Lecture 1 Introduction and preliminaries (Chapter 0) Theory of Information Lecture 1 Theory of Information Lecture 1 Introduction and preliminaries (Chapter 0)

Theory of Information Lecture 1 Introduction Theory of Information Lecture 1 Subject: Encoding data for transmission Transmission: “from here to there” (i.e., sending a message from one place to another) “from now to later” (i.e., storing data for later retrieval) Very briefly, this course is about encoding data or messages for transmission. Transmission can mean one of the two. For us this does not make difference.

Theory of Information Lecture 1 Introduction Theory of Information Lecture 1 Data encoding – representing data with strings of symbols of another alphabet (mostly, {0, 1}) 3 reasons to encode data: for efficiency (information theory) for error detection and/or correction (coding theory) for secrecy (cryptology) Such encodings are usually done separately: compression → encryption → adding bits for error correction efficiency: smaller space in storage; faster transmission. Errorrs are frequent. As a minimum, we should be able to detect, better – correct. Secrecy, so that unauthorized people cannot read the data. Our main focus will be on the first item, less on the second, and hardly any on the third. If anyone here is because of cryptology, better drop now!

Theory of Information Lecture 1 Introduction Theory of Information Lecture 1 Example of compression. How many bits would it take to encode a 100-character English word? Now assume character “a” appears 50% of the time. How many bits would it take to encode a 100-character English word?

Theory of Information Lecture 1 Introduction Theory of Information Lecture 1 Example of error detection. 1 parity bit 001000100011001011011 If errors occur, what is the probability of noticing them? How about if we used 2 parity bits? How about if we used 10 parity bits?

Theory of Information Lecture 1 Sets Theory of Information Lecture 1 Sets: {aa, bab}, {2,4,6,…} etc. Zn= {0,1,…,n-1} , or {} --- the empty set |S| --- the size (number of elements) of S bS --- b is an element of S ST --- S is a subset of T. Also, T is a superset of S. Sc --- the complement of S ST --- the union of S and T ST --- the intersection of S and T When ST=, we say that S and T are disjoint Set notation examples: {a | a is an even integer} {2k | k is an integer}

Theory of Information Lecture 1 Summation Notation Theory of Information Lecture 1 If s1,…,sn are numbers or algebraic expressions, their sum is denoted by  sk n k=1 3  2k = 21+22+23=14 k=1 3  2k = 20+21+22+23=15 k=0 4  2k = 22+23+24=28 k=2

Theory of Information Lecture 1 Permutations Theory of Information Lecture 1 A permutation of set S has two meanings: A bijective function from S to S, or An ordered arrangement of S Example: Let S={a,b,c,d} A functional permutation of S: f(a)=c, f(b)=b, f(c)=d, f(d)=a An arrangemental permutation of S: c,b,d,a, or just cbda

Theory of Information Lecture 1 Permutations Theory of Information Lecture 1 How many permutations do the following sets have? {a} {a,b} {a,b,c} How many permutations does {1,2,3,4,5} have? possibilities for the 1st element, possibilities for the 2nd element, possibilities for the 3rd element, possibilities for the 4th element, possibility for the 5th element Generally, an n-element set has permutations

Theory of Information Lecture 1 Permutations Theory of Information Lecture 1 A permutation of size k of S = = a permutation of a k-element subset of S How many permutations of size 3 does {1,2,3,4,5,6,7} have? possibilities for the 1st element, possibilities for the 2nd element, possibilities for the 3rd element, Generally, an n-element set has permutations of size k

Binomial Coefficients Theory of Information Lecture 1 Let 0  k  n. The binomial coefficient, read as “n choose k”, is defined by: n! = -------- k!(n-k)! ( ) n k THEOREM 0.1.1 A set S of size n has precisely subsets of size k. I.e. is the number of ways of choosing k elements from n elements. ( ) n k ( ) n k Why? ( ) ( ) Fact: = Why? n k n n-k

Theory of Information Lecture 1 Homework Theory of Information Lecture 1 1.1. What are the three reasons for encoding data? What are the names of the corresponding three disciplines/theories? 1.2. Write all of the subsets of the set {a,b,c}. 1.3. True or false: a) b{a,b,c} b) b {a,b,c} c) {a,c} {a,b,c} d) {a}   e) {b,a} {a,b} 1.4. a) {a,b,c}{b,d}= b) {a,b,c}{b,d}= c) Are {a,b,c} and {b,d} disjoint? d) 4  (3k+1) = k=2

Theory of Information Lecture 1 Homework Theory of Information Lecture 1 1.5. Write all the permutations of {x,y,z, t}. 1.6. How many permutations does the set {0,1,…,9} have? 1.7. Write all permutations of size 2 of the set {a,b,c,d}. 1.8. How many permutations of size 6 does the set {0,1,…,9} have? 1.9. Imagine a language that uses the 26 English characters, and where no word can contain the same character more than once. At most how many 4-character words could such a language have? 1.10. By definition, 1.11. Your class has 20 students. The teacher needs to choose 5 of them for a special assignment. How many possibilities does the teacher have? ( ) = n k