::ICS 804:: Theory of Computation - Ibrahim Otieno - iotieno@uonbi.ac.ke +254-0722-429297 SCI/ICT Building Rm. G15

Course Outline Mathematical Preliminaries Turing Machines Recursion Theory Markov Algorithms Register Machines Regular Languages and finite-state automata Aspects of Computability

Last week: Regular Languages and FSA Regular Expressions and regular languages Deterministic FSA Non-deterministic FSA Finite-state automata with epsilon moves Generative grammars Context-free, Context-sensitive languages Chomsky Hierarchy

Course Outline Mathematical Preliminaries Turing Machines Additional Varieties of Turing Machines Recursion Theory Markov Algorithms Register Machines Regular Languages and finite-state automata Aspects of Computability

Aspects of Computability Church-Turing Thesis Rice’s Theorem Wrap-Up

The Church-Turing Thesis

Church-Turing Effectively calculable function: cultural notion; part of mathematician’s informal sense of what mathematics is about Turing-computable function: not cultural or informal at all Church-Turing Thesis tries to combine these two notions (focus on number- theoretic functions)

Church-Turing Turing (1936): a number-theoretic function f is effectively calculable if and only if f is Turing-computable Not possible to formally prove it. Why not? Try to justify it Two parts: f is effectively calculable so f is Turing-computable f is Turing-computable so f is effectively calculable Obvious: there exists some algorithm, so it is calculable

Church-Turing f is effectively calculable so f is Turing- computable: not trivial Does there exist a function that is effectively calculable but not Turing- computable? Many (very different) models of computation, all related (cf. equivalence results) Turing computable functions, Markov-computable functions, … Partial Recursive Functions!

Feasibility (recap) Cobham–Edmonds thesis regarding computational feasibility: the problem of determining whether a given string is a member of a given language L is feasible if and only if L is in P Complexity class P: class of all languages accepted in polynomial time by some (single-tape) Turing Machine

Problems with the Thesis Computable by what? Focus on single-tape Turing machine Which model of computation is the favored one? Does it make a difference?

Problems with the Thesis e.g. L = {w*|na(w) = nb(w)} Register machine: O(n) steps Single Tape Turing Machine: O(n2) steps Multi-Tape Turing Machine: O(n) steps Markov Machine: O(n2) steps

Polynomial Relatedness Suppose that multitape MTM accepts language L in O(n) steps. Then there exists MRM that accepts L in time O([timeTM(n)]2) under the uniform cost assumption. Suppose that MRM accepts language L in O(n) steps under the assumption of uniform cost. Then there exists multitape MTM that accepts L in time O([timeTM(n)]3).

Complexity (recap: MA) Given Turing machine M accepting L, there exists a Markov algorithm AS that accepts L in O(timeM(n)) steps Given Markov algorithm AS accepting L, there exists a Turing machine M that accepts L in O([timeAS(n)]4) steps

Time and Space (recap: TM) Corollary Suppose that language L is accepted by k-tape Turing machine M. Then L is accepted by a single-tape Turing machine M1 such that timeM1(n) is O([timeM(n)]2) Corollary Suppose that language L is accepted by k-tape Turing machine M. Then L is accepted by a single-tape Turing machine M´ such that spaceM´(n) is O(spaceM(n)).

Results Regarding Polynomial Relatedness of Models Markov Algorithm Model O(timeM(n)) O([timeAS(n)]4) Multitape Turing Machine model O([timeTM(n)]3) O(timeM(n)) O([timeM(n)]2) Register Machine Model Single-tape Turing Machine model O([timeTM(n)]2)

Reformulation In invoking the Cobham-Edmonds thesis, it seems to make no difference which model we take Reformulation: A language-acceptance problem is feasible provided that the language in question is accepted in polynomially bounded time

More generally Invariance principle for models of sequential Computation Let Model be some machine model of sequential computation. Then Model will be said to be a reasonable model of sequential computation if it is polynomially related to the multitape Turing machine model

Church-Turing Over 7 decades, many different models of computation, but all can be related to the Turing-machine model All seem to encompass the same class of functions: partial recursive functions Hypothesis: this must be a “natural” class Mathematical properties of its members make it the natural outcome of diverse approaches This class must be the class of effectively calculable functions If one of the other models could compute some function that is not Turing-computable, then the class would not seem to be so properly defined

Church-Turing So: no formal proof, but vast amount of empirical evidence Thesis: If function f is effectively calculable, then f is Turing computable. Equivalently, if a function f is not Turing computable, then f is not effectively calculable But limits of human cognition?

Rice’s Theorem

Rice’s Theorem Preliminaries: Every natural number is the encoding or gödel number of a unique single-tape deterministic Turing machine Every Turing machine with input alphabet ={1} computes a unary partial recursive function The class of Turing machines with input alphabet ={1} can be effectively enumerated

Recursive Set (recap) A set S of natural numbers is recursive if its characteristic function S is (partial) recursive with S(n) = 1 if n  S 0 if n  S

Rice’s Theorem Theorem: Let  be a set of unary partial recursive functions that is nontrivial (i.e. not empty or the entire class). Then let  be the set of all gödel numbers of the members of . Then  is not a recursive set of natural numbers

Rice’s Theorem Important ramifications for pragmatic computer science: e.g. teacher trying to grade C programs for some assignment Approach 1: check input/output Approach 2: grading program that runs on source code to check whether it effectively computes some function Following Rice’s Theorem (and Church-Turing) there cannot be such a grading program

Wrap-up

Paradigms Three computational paradigms: Function computation Language recognition Transduction paradigm Various ways of modeling these paradigms: Turing machines, Markov algorithms, register machines

Turing Starting point: single-tape Turing machines Variations: one-way-infinite tape, halting state, multi-tape Inconsequential from theoretical point of view Variants: extensionally equivalent

Markov Algorithms Labeled and unlabeled Like Turing machines, universal model of computation Universal: Not: can do everything But: any function that is computable in any sense is computed by some Markov algorithm Church-Turing: functions that are not Markov- computable are not computable in any intuitive sense either

Computability-in-principle Computation without limitations in terms of time and resources Some functions are computable in principle, but may take ages to finish e.g. Busy Beaver is not Turing-computable

Busy Beaver The Turing Machine in TMn that writes the maximum number of 1s for a number of states Very hard to find for anything but the smallest of n’s For any n: large number of possible Turing Machines Some Turing Machines will loop (so (n) = 0) Hard to distinguish between infinitely looping machine and one that is just writing a lot of 1s

Busy Beaver Calculated by brute force

Busy Beaver Very theoretic, toy-like argument to non- computability But: impact on pragmatic computer science as well Rice’s theorem: various decision problems are not solvable by any Turing machine and thus not solvable at all Various algorithms for proving program correctness do not exist: computer scientists with no background in theory can waste a lot of time trying to develop such an algorithm

Theory of Computation Origins in 1920s/1930s by mathematical logicians and philosophers before the advent of the modern digital computer!!! Theoretic computer science has been advancing independently from pragmatic computer science But: theoretic work has influenced pragmatic developments and vice versa e.g. Universal Turing Machine as a precursor to modern-day Computability theory influences design of programming languages and compilers Register machines: reflect modern machine architecture

Theory of Computability A side-branch of mathematics But: mathematicians cannot axiomatically characterize mathematical truth Church-Turing, Cobham-Edmonds, … try to define computability and seem to have largely succeeded Applications in pragmatic computer science have validated these insights But: theory of computation is a science in its own right (e.g. theoretical physics, theoretical linguistics, cognitive science, …) What is computable? What is feasible? Basic philosophical questions!

Feasible Computation More interesting from a practical point of view Complexity theory can aid during program design Cobham-Edmonds Thesis: feasibility is modeled by a deterministic, single-tape Turing machine whose running time is bounded above by some polynomial in the size of the input Complexity class P (and PNP): can be computed in polynomial time Complexity class NP: requires exponential time NP-complete problems not in the class of P are deemed to be computationally unfeasible

Feasible Computation So when working on a problem: if you find the solution to not be solvable in polynomial time, direct your effort to a subset of the problem E.g. traveling salesman problem for Weighted undirected graphs  refocus the problem with some heuristics, e.g. nearest-neighbor traveling salesman problem So complexity theory has had a huge impact outside of theoretical computing

Feasible Computation P-completeness: although computation is feasible, the problem is unlikely to be solvable using a parallel computing approach in other words: a parallel algorithm will not produce any improvement in worst-case execution time over the polynomially bounded non-parallel algorithm that already exists So knowing this: if a problem is in complexity class P and is polynomially bounded in time, don’t waste time trying to parallelize the solution

Part II: The Chomsky Hierarchy Accepted by nondeterministic push-down stack automaton G: Context-Free Grammars Accepted by linear-bounded automaton G: context-sensitive grammars Accepted by deterministic FSA G: Right-linear grammars Context-free languages Recursively enumerable languages Context- sensitive languages Regular languages Accepted by deterministic 1tape Turing Machine

Open questions P = NP ? Can deterministic linear-bounded automaton accept the same language as a nondeterministic one? Nondeterminism, although proven to be inconsequential for Turing Machines and finite- state automata, is still not fully understood for models of computation