1 Theory of Computation Introduction Text Book: Michael Sipser. Introduction to the Theory of Computation. 2nd ed. Boston, MA: Thomson Course Technology, ISBN:

2 What is Automata Theory? Study of abstract computing devices, or “machines” Automation = an abstract computing device Note: A “device” need not even be a physical hardware! A fundamental question in computer science: Find out what different models of machines can do and cannot do The theory of computation Computability vs. Complexity

3 An Abstract Machine To make a more solid definition of algorithm we need to define an abstract (general) machine which can perform any algorithm that can be executed by any computer. Then, We need to show that indeed this machine can run any algorithm that can be executed by any other computer. Then, 1. We can associate the notion of algorithm with this abstract machine. 2. We can study this machine to find the limitations of computations. (Problems with no computation available to solve.)

4 A simple «computer» BATTERY SWITCH off on start f f input: switch output: light bulb actions: f for “flip switch” states: on, off bulb is on if and only if there was an odd number of flips

5 Another computer BATTERY off start 1 inputs: switches 1 and 2 actions: 1 for “flip switch 1” actions: 2 for “flip switch 2” states: on, off off on bulb is on if and only if both switches were flipped an odd number of times

6 A design problem 4 BATTERY ? Can you design a circuit where the light is on if and only if all the switches were flipped exactly the same number of times?

7 A design problem Such devices are difficult to reason about, because they can be designed in an infinite number of ways By representing them as abstract computational devices, or automata, we will learn how to answer such questions They can describe the operation of any “small computer”, like the control component of an alarm clock or a microwave They are also used in lexical analyzers to recognize well formed expressions in programming lang ab1 is a legal name of a variable in C 5u= is not

8 Different kinds of automata This was only one example of a computational device, and there are others We will look at different devices, and look at the following questions: What can a given type of device compute, and what are its limitations? Is one type of device more powerful than another?

9 Some devices we will see finite automataDevices with a finite amount of memory. Used to model “small” computers. push-down automata Devices with infinite memory that can be accessed in a restricted way. Used to model parsers, etc. Turing MachinesDevices with infinite memory. Used to model any computer. Time-bounded Turing Machines Infinite memory, but bounded running time. Used to model any computer program that runs in a “reasonable” amount of time.

10 Computation

11 Theory of Computation A very fundamental and traditional branch of Theory of Computation seeks: 1. A more tangible definition for the intuitive notion of algorithm which results in a more concrete definition for computation. 2. Finding the boundaries (limitations) of computation. We want to answer fundamental questions: What can we do in a reasonable amount of time? What can we do in a reasonable amount of space? What can we do at all?

12 Course Outlines Introduction: Mathematical preliminaries: sets, relations, functions,sequences, graphs, trees, proof by induction, definition by induction (recursion). Basics of formal languages: alphabet, word, sentence, concatenation,union, iteration [= Kleene star], language, infinity of languages, finite representations of languages

13 Course Outlines PART I: Finite Automata and Regular Sets DFA,NFA,regular expressions and their equivalence Limitation of FAs; Closure properties of FAs, Optimization of FAs PART II: Pushdown Automata and Context Free Languages CFGs and CFLs; normal forms of CFG Limitation of CFG; PDAs and their variations, closure properties of CFLs Equivalence of pda and CFGs; deterministic PDAs parsing (Early or CYK's algorithms)

14 Course Outlines PART III: Turing Machines and Effective Computability Turing machine [& its variations] and Equivalence models Universal TMs Decidable and undecidable problems (Recursive sets and recursively enumerable sets) Problems reductions ; Some undecidable problems

15 Course Outlines PART III: Turing Machines and Effective Computability Turing machine [& its variations] and Equivalence models Universal TMs Decidable and undecidable problems (Recursive sets and recursively enumerable sets) Problems reductions ; Some undecidable problems

16 Goals Designing a new programming language for a specialized application Context Free Languages Dealing with string searching and pattern matching Finite Automata and regular expressions Confronted with a problem that seems to require more computer time than you can afford NP-Completeness «What are the fundemantal capabilities and limitations of computers?»

17 Goals Finite automata We will understand what kinds of things a device with finite memory can do, and what it cannot do Introduce simulation: the ability of one device to “imitate” another device Introduce nondeterminism: the ability of a device to make arbitrary choices Push-down automata These devices are related to grammars, which describe the structure of programming (and natural) languages

18 Goals Turing Machines This is a general model of a computer, capturing anything we could ever hope to compute Surprisingly, there are many things that we cannot compute, for example: It seems that you should be able to tell just by looking at the program, but it is impossible to do! Write a program that, given the code of another program in C, tells if this program ever outputs the word “ hello ”

19 Alan Turing ( ) Father of Modern Computer Science English mathematician Studied abstract machines called Turing machines even before computers existed Heard of the Turing test? (A pioneer of automata theory)

20 Turing Machine A conceptual model for general purpose computers proposed by Alan Turing in A Turing machine has an unlimited and unrestricted amount of memory. A Turing machine can do everything a real computer can do. Nevertheless there are problems that a Turing machine cannot solve. In a real sense, these problems are beyond the theoretical limits of computations.

21 Theory of Computation: A Historical Perspective 1930s Alan Turing studies Turing machines Decidability Halting problem s “Finite automata” machines studied Noam Chomsky proposes the “Chomsky Hierarchy” for formal languages 1969 Cook introduces “intractable” problems or “NP-Hard” problems Modern computer science: compilers, computational & complexity theory evolve

22 Languages & Grammars Or “words” Image source: Nowak et al. Nature, vol 417, 2002 Languages: “A language is a collection of sentences of finite length all constructed from a finite alphabet of symbols” Grammars: “A grammar can be regarded as a device that enumerates the sentences of a language” - nothing more, nothing less N. Chomsky, Information and Control, Vol 2, 1959

23 The Central Concepts of Automata Theory

24 Alphabet An alphabet is a finite, non-empty set of symbols We use the symbol ∑ (sigma) to denote an alphabet Examples: Binary: ∑ = {0,1} All lower case letters: ∑ = {a,b,c,..z} Alphanumeric: ∑ = {a-z, A-Z, 0-9} DNA molecule letters: ∑ = {a,c,g,t} …

25 Strings A string or word is a finite sequence of symbols chosen from ∑ Empty string is  (or “epsilon”) Length of a string w, denoted by “|w|”, is equal to the number of (non-  ) characters in the string E.g., x = |x| = 6 x = 01  0  1  00  |x| = ? xy = concatentation of two strings x and y

26 Powers of an alphabet Let ∑ be an alphabet. ∑ k = the set of all strings of length k ∑* = ∑ 0 U ∑ 1 U ∑ 2 U …(the set of all strings over  ∑ + = ∑ 1 U ∑ 2 U ∑ 3 U …

27 Languages L is a said to be a language over alphabet ∑, only if L  ∑*  this is because ∑* is the set of all strings (of all possible length including 0) over the given alphabet ∑ Examples: 1. Let L be the language of all strings consisting of n 0’s followed by n 1’s: L = { ,01,0011,000111,…} 2. Let L be the language of all strings of with equal number of 0’s and 1’s: L = { ,01,10,0011,1100,0101,1010,1001,…} Definition:Ø denotes the Empty language Let L = {  }; Is L=Ø? NO Canonical ordering of strings in the language

28 Examples of Practical Formal Languages Ex: Let D be the set of all ASCII codes.  a C program is simply a finite string over D satisfying all syntax rules of C.  C-language = def { x | x is a well-formed C program over D }.  PASCAL-language = {x | x is a well-formed PASCAL program over D }. Similarly, let ENG-DIC =The lexicon of all English words = { John, Mary, is, are, a, an, good, bad, boys, girl,..}  an English sentence is simply a string over ENG-DIC  ==> English = def {x | x is a legal English sentence over END-DIC} ==>  1.John is a good boy.  English.  2. |John is a good boy. | = ?

29 Issues About Formal Languages  Why need formal languages?  for specification (specifying programs, meanings etc.)  i.e., basic tools for communications b/t people and machines.  although FL does not provide all needed theoretical framework for subsequent (semantic processing...) processing, it indeed provides a necessary start, w/t which subsequent processing would be impossible -- first level of abstraction.  Many basic problems [about computation] can be investigated at this level.  How to specify(or represent) a language ?  Notes: All useful natural or programming languages contain infinite number of strings (or programs and sentences)

30 How to specify a language?  principles: 1. must be precise and no ambiguity among users of the language: 2. efficient for machine processing  tools:  1. traditional mathematical notations:  A = {x | |x| < 3 and x  {a,b}} = {e,a,b,aa,ab,ba,bb}  problem: in general not machine understandable.  2. via programs (or machines) :  P: a program; L(P) = def {x | P return 'ok' on input string x}  precise, no ambiguity, machine understandable.  hard to understand for human users !!  3. via grammars: (easy for human to understand)  Ex: noun := book | boy | John | Mary art := a | an | the ; prep := on | under | of |... adj := good | bad | smart |... NP := noun | art noun | NP PP |... PP := prep NP ==> 'the man on the bridge'

31 Operations on strings  string concatenations:  x,y: two strings ==> x·y is a new string with y appended to the tail of x. i.e., x·y is the function :  z : [0, len(x)+len(y) )   such that  z(n) = x(n) for 0  n < len(x) and  z(len(x)+n) = y(n) for 0  n < len(y).  Some properties of · : 1. ASSOC: (xy)z = x(yz) ; 2. Identity:  x = x  = x. 3. |xy| = |x| + |y|.  conventions and abbreviations:  S: for alphabet ; a,b,c: for symbols;  x,y,z: for strings; A,B,C: for languages;  x 5 for xxxxx; x 1 = x ; x 0 = .  #a(x) = def number of a's in x. ==> #a(aabbcca) = 3.

32 Operations on strings 1. usual set operations:  Union: A  B = {x | x  A or x  B } Ex: {a,ab} U { ab, aab} = {a,ab,aab}  intersection: A  B = {x | x  A and x  B }  complements in  *: ~A = def  * - A = { x | x not  A}  ex: ~{x | |x| is even } = {x | |x| is odd }. 2. Set concatenations: A·B = def {xy | x  A and y  B }.  Ex: {b,ba} {a,ab} = {ba,bab,baa,baab}. 3. Powers of A: A n ( n ³ 0) is defined inductively: A 0 = {  }; A n+1 = A·A n = A·A·...·A n A's

33 Operations on strings Ex: Let A = {ab,abb}. Then 1. A 0 = ? 2. A 1 = ?3. A 2 = ?4. |A 4 |=? 5. Hence {a,b,c} n = {x  {a,b,c}* | |x| = n } and A n = { x 1 x 2...x n | x 1,...,x n  A } 4. Asterate (or star) A* of A is the union of all finite powers of A: A* = def U k  0 A K = A 0 U A UA 2 U A 3 U... = {x 1 x 2...x n | n  0 and x i  A for 1  i  n } notes: 1. n can be 0 ==>   A*. ==>   {}*. 2. If A =  ==> A* =  * = the set of all finite strings over . 5. A + = def the set of all nonzero powers of A = def U k  1 A k = A U A 2 U A 3 U... = A A*.

34 The Membership Problem Given a string w  ∑*and a language L over ∑, decide whether or not w  L. Example: Let w = Q) Is w  the language of strings with equal number of 0s and 1s?

35 Finite Automata Some Applications Software for designing and checking the behavior of digital circuits Lexical analyzer of a typical compiler Software for scanning large bodies of text (e.g., web pages) for pattern finding Software for verifying systems of all types that have a finite number of states (e.g., stock market transaction, communication/network protocol)

36 Structural expressions Grammars Regular expressions E.g., unix style to capture city names such as “Palo Alto CA”: [A-Z][a-z]*([ ][A-Z][a-z]*)*[ ][A-Z][A-Z] Start with a letter A string of other letters (possibly empty) Other space delimited words (part of city name) Should end w/ 2-letter state code

37 Formal Proofs

38 Deductive Proofs From the given statement(s) to a conclusion statement (what we want to prove) Logical progression by direct implications Example for parsing a statement: “If y≥4, then 2 y ≥y 2.” (there are other ways of writing this). given conclusion

39 Example: Deductive proof Let Claim 1: If y≥4, then 2 y ≥y 2. Let x be any number which is obtained by adding the squares of 4 positive integers. Claim 2: Given x and assuming that Claim 1 is true, prove that 2 x ≥x 2 Proof: 1) Given: x = a 2 + b 2 + c 2 + d 2 2) Given: a≥1, b≥1, c≥1, d≥1 3)  a 2 ≥1, b 2 ≥1, c 2 ≥1, d 2 ≥1(by 2) 4)  x ≥ 4(by 1 & 3) 5)  2 x ≥ x 2 (by 4 and Claim 1) “implies” or “follows”

On Theorems, Lemmas and Corollaries We typically refer to: A major result as a “theorem” An intermediate result that we show to prove a larger result as a “lemma” A result that follows from an already proven result as a “corollary” 40 An example: Theorem: The height of an n-node binary tree is at least floor(lg n) Lemma: Level i of a perfect binary tree has 2 i nodes. Corollary: A perfect binary tree of height h has 2 h+1 -1 nodes.

41 Quantifiers “For all” or “For every” Universal proofs Notation * =? “There exists” Used in existential proofs Notation * =? Implication is denoted by => E.g., “IF A THEN B” can also be written as “A=>B” * I wasn’t able to locate the symbol for these notation in powerpoint. Sorry! Please follow the standard notation for these quantifiers. These will be presented in class.

42 Proving techniques By contradiction Start with the statement contradictory to the given statement E.g., To prove (A => B), we start with: (A and ~B) … and then show that could never happen What if you want to prove that “(A and B => C or D)”? By induction (3 steps) Basis, inductive hypothesis, inductive step By contrapositive statement If A then B ≡If ~B then ~A

43 Proving techniques… By counter-example Show an example that disproves the claim Note: There is no such thing called a “proof by example”! So when asked to prove a claim, an example that satisfied that claim is not a proof

44 Different ways of saying the same thing “If H then C”: i. H implies C ii. H => C iii. C if H iv. H only if C v. Whenever H holds, C follows

45 “If-and-Only-If” statements “A if and only if B” (A B) (if part) if B then A ( <= ) (only if part) A only if B ( => ) (same as “if A then B”) “If and only if” is abbreviated as “iff” i.e., “A iff B” Example: Theorem: Let x be a real number. Then floor of x = ceiling of x if and only if x is an integer. Proofs for iff have two parts One for the “if part” & another for the “only if part”