1. Theory of Computation CS3231, 2010-2011 First Semester Rahul Jain TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A AAAA A

2. Why do I care about Theory ? It provides solid foundations. Provides stability : Technologies keep changing, theory does not. Its beautiful and elegant. It expands your mind. It is useful as well ! It is relevant to practice. Finally: Aren’t you curious about the ultimate capabilities and limitations of computers ? Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 2

3. Computability Theory What are the fundamental capabilities and limitations of computers ? What can(not) computers solve ? Complexity Theory What can be solved given limited resources of time and space ? In this course we will only worry about Computability Theory (in practice the Chapters 0-5 of Sipser). 3 Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain Two main parts of Theory

4. Chapter Mathematical Preliminaries Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 4

5. Mathematical Notions and Terminology Sets : Set, element (a.k.a member), subset, proper subset, power set, multiset, infinite set, natural numbers, integers, empty set, union of sets, intersection of sets, complement of a set, Venn diagrams. Sequences and Tuples : Sequence, k-tuple, pair, Cartesian product (a.k.a. cross product). Functions and Relations : Function (a.k.a. mapping), domain, range, onto function, arguments of a function, k-ary function, unary function, binary function, infix notation, prefix notation, predicate (a.k.a. property), relation, k-ary relation, binary relation, equivalence relation (reflexive, symmetric and transitive), Ex. 0.11. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 5

6. Mathematical Notions and Terminology Graphs : Undirected graph, nodes (a.k.a. vertices), edges, degree of a node, labeled graph, subgraph, path, simple path, connected graph, cycle, simple cycle, tree, root of a tree, leaves of a tree, directed graph, outdegree, indegree, directed path, strongly connected graph, consider examples, Ex. 0.17 done in class. Strings and Languages : Alphabet, symbols of alphabet, string over an alphabet, length of a string, empty string, reverse of a string, substring, concatenation, lexicographic ordering of strings, language, consider examples. Boolean Logic : Boolean logic, boolean values (True and False), boolean operations, negation (a.k.a. NOT), conjunction (a.k.a. AND), disjunction (a.k.a. OR), operands of an operation, exclusive OR (a.k.a XOR), equality, implication, distributive law. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 6

7. Definitions, Theorems, and Proofs Basics: Definitions, mathematical statements, proof, theorem, lemmas, corollaries. Finding Proofs : Multipart statement e.g. if and only if (a.k.a. iff), forward direction, backward direction, counterexample. Tips for producing a proof : 1.Be patient. 2.Come back to it. 3.Be neat. 4.Be concise. Theorem 0.20 (DeMorgan’s Law) : Proof done in class. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 7

8. Definitions, Theorems, and Proofs Types of Proofs : 1.Proof by construction : Theorem 0.22 : For each even number n greater than 2, there exists a 3- regular graph with n nodes. Proof done in class. 2.Proof by contradiction : Theorem 0.24 : Square root of 2 is irrational. Proof done in class. 3.Proof by induction : Useful for proving statements for infinite sets; Basis step, induction step, induction hypothesis, strong induction. Theorem 0.25 : Calculating the principal amount after t-months given the rate of interest and monthly installments. Proof done in class. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 8

9. Three types of Computers we will study Finite Automaton : Computers with finite memory. Pushdown Automaton : Computers with infinite memory which can only be used as a stack. Turing Machines : Computers with infinite memory which can be accessed sequentially. The most general computers according to Church-Turing Thesis. (In effect we will study Chapters 0-5 of Sipser). Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 9

10. Chapter Finite Automaton Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 10

11. Deterministic Finite Automaton (DFA) Example of revolving door, example of a state diagram M2. Formal definition of finite automaton : Set of states, alphabet, transition function, start state, set of accepting states (a.k.a final states). Definition 1.5 (Deterministic Finite Automaton (DFA) ): Roughly speaking, a string w is said to be accepted by DFA M, if starting from the start state and following the transitions of the DFA, the string w reaches one of the final states. Formal description of computation (done in class). Language recognized by DFA M is the set of strings accepted by M. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 11

12. Deterministic Finite Automaton (DFA) Regular languages (Definition 1.16 ) : A language is called a regular language if some DFA recognizes it. More examples : Formal description of M2, language of M2, Examples. 1.9, 1.11, 1.13, 1.15, 1.17 (done in class). Designing Finite Automaton for a language: 1.Use “Reader as automaton” idea. 2.Determine what you need to remember. This should be finite information. 3.Assign a state of each of the possibilities. 4.Determine transitions between states. 5.Determine start state and set of accepting states. Fig 1.18 example, example 1.21 (done in class). Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 12

13. Non-Deterministic Finite Automaton (NFA) Example 1.30, consider computation tree on example 100 and 011. Example 1.33 (all done in class). Formal Definition of NFA : Definition 1.37 (highlight differences with DFA): Roughly speaking, an NFA M accepts a string w, if (among many possible valid sequences of transitions on w, starting from the start state) there exists one valid sequence, which reaches one of the final states. Formal definition of computation for NFA done in class. Example 1.38 (done in class). The language recognized by NFA M is the set of all strings accepted by M. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 13

14. Equivalence of NFA and DFA Def. (Equivalent machines) : Two machines are called equivalent if they recognize the same language. Theorem : Every NFA is equivalent to an NFA with a single accept state. Proof in class. Idea : Choose one of the accept states q. Give epsilon transitions from all other accept states to q and remove them from the set of accept states. Every DFA is also an NFA (easy to argue, DFA is just a special case of NFA). Theorem 1.39 : Every NFA has an equivalent DFA. Proof in class. Idea : Keep account of all possible states the NFA could be in after every transition, including epsilon transitions. Corollary 1.40 : A language is regular if and only if some NFA recognizes it. Example 1.41 (done in class). Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 14

15. Closure under Regular Operations Definition 1.23 (Regular operations): A set being closed under a given operation (e.g. Natural numbers closed under addition, multiplication and not division). Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 15

16. Closure under Regular Operations Theorem (Closure of regular languages under regular operations) : Class of regular languages is closed under : 1.Union. 2.Concatenation. 3.Star operation. Proof done in class. Idea : Next slide. In addition regular languages are also closed under : 4.Complement. 5.Intersection. Proof done in class. Idea : For complement, given a DFA, just make accept states non-accept and non-accept states as accept. For intersection use De-Morgan’s Law. Theorem : Every finite language L is regular. Proof done in class. Idea : For any string w, it is easy to see that {w} is a regular language (easy to make an NFA for it). Now take finite union to get the finite set L. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 16

17. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 17

18. Regular Expressions Definition 1.52 (Regular expressions) : Language described by a regular expression. Order of operations : star, concatenation and then union (unless specified otherwise by parenthesis). Example 1.53 (some parts done in class, specially parts 11. and 12. since they describe some conventions.) Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 18

19. Regular Expressions Equivalence with Finite Automaton : Theorem : The language described by a regular expression is regular. Proof done in class. Idea : Construct NFA successively by constructing NFA for each part. Examples 1.56 and 1.58 done in class. Theorem : Every regular language can be described by a regular expression. Proof done in class : Define first Generalized Non-Deterministic Finite Automaton (GNFA). Figure 1.61. Give formal definition (Definition 1.64). Describe how to convert GNFA to regular expression by reducing the number of states to 2, removal of a given state (Figure 1.63). Examples 1.66, 1.68 (done in class). Corollary : A language is regular if and only if some regular expression describes it. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 19

20. Non-regular languages Important to be careful (both below seem to require unbounded memory) 1.C = {w | w has an equal number of 0s and 1s}. 2.D = {w | w has an equal occurrences of 01 and 10 as substrings}. C is not regular, however D is regular ! Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 20

21. Non-regular languages Theorem (Pumping Lemma) : Proof done in class. Idea : Let A have DFA M. Let p be the number of states in M. Then any string s in A, of length at least p will go through a loop in M and this loop part can be pumped up or down to get more strings in A (Fig 1.71, 1.72). Examples 1.73, 1.74, 1.75, 1.76, 1.77 (all done in class.) Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 21

22. Chapter Context Free Languages and Pushdown Automaton Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 22

23. Context Free Languages Context free grammars (CFG) quite useful for parsers in compilers. CFG : Substitution rules (a.k.a. productions), variables, terminals, start variable, derivation, yields, derives, language of a grammar, parse tree for a string in grammar, Context Free languages (CFL) (a language generated by a CFG). Do example to illustrate these concepts. Formal Definition (CGF) : Examples 2.3, 2.4. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 23

24. Context Free Languages Designing CFG : 1.Break into pieces if possible. Do example. 2.Creating CFG from a DFA. 3.Take care of links between parts of strings. 4.Take care of recursive structure. Ambiguous Grammar : Do example Def (Ambiguous Grammar): A string w is derived ambiguously in context-free grammar G if it has two or more different leftmost derivations (at every step the leftmost remaining variable is replaced). Grammar G is ambiguous if it generates some string ambiguously. Inherently ambiguous languages : Context-Free Languages that can only be generated by ambiguous grammars. E.g. {a^ib^jc^k | i = j or j = k } is inherently ambiguous. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 24

25. Context Free Languages Chomsky normal form : Quite useful in giving algorithms for working with CFGs. Theorem : Any CFL is generated by a CFG in Chomsky normal form. Proof done in class. Idea : 1.Add a new start variable. 2.Remove all epsilon rules of the form A → ε. 3.Remove all unit rules of the form A → B. 4.Convert the remaining rules in proper form. Example 2.10. done in class. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 25

26. Pushdown Automaton Pushdown automaton (PDA) are like finite automaton with additional component called stack (last in first out storage). Schematic of finite automaton, Schematic of Pushdown automaton (done in class). Pushing a symbol, popping a symbol. Do informal description of PDA for the language {0 n 1 n | n ≥ 0}. Deterministic and Non-Deterministic PDA are not equivalent in power. Non-Deterministic PDA are equivalent to Context Free Grammars (we only focus on them). Formal Definition : Formal Description of when PDA accepts a given string w (done in class). Language of a PDA M is the set of strings accepted by M. Examples 2.14, 2.16 (needs non-determinism), 2.18 done in class (testing for end of stack, testing for end of input). Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 26

27. Pushdown Automaton Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 27 Equivalence with Context free grammars Theorem: If a language is context free, then some pushdown automaton recognizes it. Proof Idea : To simulate CFG G, construct PDA P as follows: 1.Place the $ and start variable S on the stack. 2.Repeat the following forever : 1.If the top of the stack is a variable A, non-deterministically select one of the rules for A and substitute A by the string on the right-hand side of the rule. 2.If the top of stack is a terminal symbol ‘a’, read the next symbol from the input and compare it to ‘a’. If they match, repeat. If not reject. 3.If the top of the stack is $, enter accept state. Figure 2.24. gives the state diagram of PDA.

28. Pushdown Automaton Equivalence with Context free grammar Theorem : If a pushdown automaton recognizes some language, then it is context free. Proof Idea : Want to convert a PDA P to CFG G. First assume w.l.og. that P has a single accept state, it empties the stack before accepting, and each transition has either a push or a pop but not both, nor neither. Claim 1 : If A pq generates x, then x can bring P from p with empty stack to q with empty stack. Claim 2 : If x can bring P from p with empty stack to q with empty stack, A pq generates x. Proofs done in class. Corollary : Every regular language is context free. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 28

29. Non-Context free languages Pumping lemma for context-free languages : Proof done in class. Idea : Consider the grammar for A. Any sufficiently long string will have variables repeated in its derivation. This repeated part can be pumped up or down to get other strings in A. Examples 2.36, 2.37, 2.38, done in class. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 29

30. Chapter Turing Machines Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 30

31. Turing Machines Proposed by Alan Turing in 1936. Schematic diagram of turing machine M (explain its functioning). 1.A turing machine can both write on the tape and read from it. 2.The read write head can move both left and right. 3.The tape is infinite. 4.The special accept and reject states take effect immediately. Example of {w#w : w in {0,1}*}. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 31

32. Turing Machines Configuration of turing machine uqv: 1.uv is the current tape content. 2.q is the current state. 3.Current head location is the first symbol of v. 4.The tape contains only blank symbols after the last symbol of v. Configurations yielding to each other (do examples), start configuration on input w: (q 0, w), accepting configuration, rejecting configuration, halting configuration, when does M accept w. Definitions : 1.Language recognized by M (a.k.a language of M) denoted L(M) : The set of all strings accepted by M. M is called recognizer for L. 2.Turing recognizable language (a.k.a. recursively enumerable languages): A language L is called turing recognizable if there exists a turing machines M such that L = L (M). 3.Decider : A turing machine that halts on all inputs (loops on no input). A decider that recognizes L is said to decide L. 4.Turing decidable language (a.k.a recursive language): A language is said to be turing decidable or just decidable if there exists a turing machine that decides it. Every decidable language is recognizable. Examples 3.7, 3.9, 3.11, 3.12. Terminology for describing Turing machines : Encoding of inputs. Formal description. Implementation level description and high-level description. Example 2.23. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 32

33. Variants of Turing Machines 1.Stay put feature. 2.Multitape turing machines. Definition done in class. Thm: Every mutitape turing machine has an equivalent single-tape turing machine. Proof : Done in class. Idea : Simulate all the tapes using single tape. (Figure 3.14.) Corollary : A language is Turing recognizable if and only if some multitape turing machine recognizes it. 3.Nondeterministic turing machines. Definition done in class. Thm: Every nondeterministic turing machine has an equivalent deterministic turing machine. Proof : Done in class. Idea: Simulate all branches of nondeterminism and accept if any branch accepts. (Figure 3.17). Corollary: A language is Turing recognizable if and only if some nondeterministic turing machine recognizes it. A nondeterministic turing mahcine is called a decider if it halts on all its branches of computation. Corollary: A language is decidable if and only if some nondeterministic turing machine decides it. 4.Enumerators : Definition done in class. Figure 3.20. Thm: A language is Turing recognizable if and only of some enumerator enumerates it. Proof: Done in class. 5.It has been shown to be equivalent with many other models of computation e.g. λ-calculus proposed by Alonzo Church (1936). Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 33

34. What is an Algorithm ? The Church-Turing Thesis Intuitive notion of algorithms = Turing Machine algorithms. All ‘reasonable’ definitions (requiring finite amount of work to be done in a single step) of algorithms are equivalent. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 34 Hilbert’s 10 th Problem (among the 23 problems posed in 1900): Give an algorithm (in Hibert’s words “a process according to which it can be determined by a finite number of operations“) for {p | p is a multi-variate polynomial with an integral root}. David Hilbert ‘assumed’ that an algorithm for the language above exits. In 1970, Yuri Matijasevic, building on the work of Marin Davis, Hilary Putnam and Julia Robinson, showed that the above problem is NOT Turing Decidable. So assuming Church-Turing Thesis, there is no algorithm for Hilbert’s 10 th problem. It is easily seen that the above language is recognizable. The language {p | p is a single variate polynomial with an integral root } is decidable. The roots of such polynomial must lie in the interval [-kc max /c 1, -k cmax /c 1 ]. Informally speaking an algorithm is a collection of simple instructions for carrying out some tasks.

35. Chapter Decidability Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 35

36. Decidability The following languages are decidable. is undecidable. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 36

37. Undecidability Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 37 The Diagonalization method : Introduced by Georg Cantor in 1873. He wanted to compare sizes of infinite sets. One-to-one functions, onto functions, correspondence. Two sets are said to have the same size iff there exists a correspondence between them. Definition : A set A is countable if either it is finite or it has the same size as N. A set which is not countable is called uncountable. Do examples 4.13, 4.15. Thm : R is uncountable. Proof : Done in class, use diagonalization method. Corollary : Some (almost all) the languages (over any alphabet) are not Turing recognizable. Proof : The set of all Turing machines is countable. The set of all languages (over any alphabet) is uncountable.

38. Undecidability Thm: A TM is Turing-recognizable. A TM = { | M is a TM that accepts string w } is undecidable. Proof: Done in class, for first part describe Universal Turing Machine. For second part use diagonalization method. Def: We say that a language is co-Turing-recognizable if it is the complement of a Turing-recognizable language. Thm: A language is decidable iff it is Turing recognizable and co-Turing recognizable. Proof: Done in class. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 38

39. Chapter Reducibility Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 39

40. Reducibility The following languages are undecidable : Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 40

41. Mapping Reducibility Definition (Computable function) : Definition (Mapping reducibility) : Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 41

42. Mapping Reducibility Examples 5.24, 5.26, 5.27. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 42

43. Computation history method Accepting computation history of machine M on input w : A finite sequence of configurations C 1, C 2, …, C l, where C 1 is the start configuration of M on w, C l is an accepting configuration of M, and each C i follows legally from C i-1 according to the rules of M. Rejecting computation history : Similarly defined accept C l is a rejecting configuration. Linear Bounded Automaton : Restricted type of Turing machine where space used on input w is |w| (does not move off the portion of the tape containing the input). By using larger alphabet we can think that the space used is constant times |w| (that is why the name linear). Examples : A DFA, A CFG, E DFA, E CFG can be decided by LBAs. Every CFL can be decided by LBAs. Lemma : Let M be an LBA with q states and g symbols in the tape alphabet. There are exactly qng n distinct configurations of M for a tape of length n. Proof done in class. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 43

44. Computation history method Theorem : A LBA = { | M is an LBA that accepts string w} is decidable. Proof done in class. Use previous lemma. Theorem : E LBA = { | M is an LBA where L(M) = φ} is undecidable. Proof done in class. Use computation history method. Theorem : ALL CFG = { | G is a CFG and L(G) = Σ*} is undecidable. Proof done in class. Use computation history method. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 44

45. Post Correspondence Problem (PCP) Define domino, collection of dominos, collection of dominos with a match. PCP = { | P is a collection of dominos which has a match.} MPCP = { | P is a collection of dominos which has a match starting with the first domino.} (MPCP = Modified PCP) Theorem : MPCP is undecidable. Proof done in class. Reduce A TM to MPCP via accepting computation histories. Theorem : PCP is undecidable. Proof done in class. Reduce MPCP to PCP. Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 45

46. The END Good Luck ! Theory of Computation, CS3231, Semester 1, 2010-11, Lecturer: Rahul Jain 46