CS 350 — Fall 2018 gilray.org/classes/fall2018/cs350/ The decision problem (entscheidungsproblem), halting problem, & Turing machines CS 350 — Fall 2018 gilray.org/classes/fall2018/cs350/

Wir müssen wissen; wir werden wissen! …We must know; we will know!” “We must not believe those, who today, with philosophical bearing and deliberative tone, prophesy the fall of culture and accept the ignorabimus [i.e., that fundamental ignorance and limitation underly reason, science, logic]. For us there is no ignorabimus, and in my opinion none whatever in natural science. In opposition to the foolish ignorabimus our slogan shall be: Wir müssen wissen; wir werden wissen! …We must know; we will know!” David Hilbert

entscheidungsproblem (decision problem): Hilbert’s entscheidungsproblem (decision problem): What general process (i.e., algorithm) will decide if a given proposition (in standard first-order logic, or a similar system) is provable/deducible, or not? (Surely such a process exists!) 1928 (generalizes Hilbert’s 10th problem from his famous 23 problems for the turn of the century in 1900) David Hilbert

But, there are undecidable problems… Alan Turing Alonzo Church

Decision procedures A decision procedure is some algorithm resulting in a Yes or No decision—True or False. A decision procedure corresponds to determining if a string is a member of a formal language, or, equivalently, if a proposition holds: is some statement true. The entscheidungsproblem asks specifically for a decision procedure for all statements written in first-order logic. Other examples of decision procedures: Is the string w accepted by the given NFA? Is the string w accepted by the given PDA? Is the string w generated by the given CFG? Is the string w in the language L?

Undecidable problems Some problems are decidable, there exists a decision procedure that always correctly yields True or False (e.g., a string’s membership in a regular language), but some are not. E.g. “Is the string w in the language L?” is undecidable when L is the language of all C programs without buff. overflow. L is the language of all valid and true english statements. L is the language of all (textually encoded) Conway’s game of life states following a particular setup.

Conway’s game of life At each iteration: cells with three living neighbors are alive; cells with two remain as they were; all other cells die.

Undecidable problems Some problems are decidable, there exists a decision procedure that always correctly yields True or False (e.g., a string’s membership in a regular language), but some are not. E.g. “Is the string w in the language L?” is undecidable when L is the language of all C programs without buff. overflow. L is the language of all valid and true english statements. L is the language of all (textually encoded) Conway’s game of life states following a particular setup. L is the language of Python programs that halt.

The Collatz (“3n+1”) conjecture Start with any positive integer, n. If even, the number is modified by dividing it in half to obtain n/2; or, if odd, the number is modified by multiplying it by three and adding one to obtain 3n+1. Repeat. Eventually, n will become 1. E.g., 8, 4, 2, 1, 4, 2, 1, 4, 2, 1, 4, 2, 1, … E.g., 9, 28, 14, 7, 22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5,16, 8, 4, 2, 1, 4, 2, 1, 4, 2, 1, 4, 2, 1, …

def collatz(n): while n != 1: if n%2 == 0: n = n/2 else: n = 3*n+1 Collatz and the halting problem def collatz(n): while n != 1: if n%2 == 0: n = n/2 else: n = 3*n+1 Would this definition for collatz(n) be in the language of Python programs that must halt? Intuitively: the halting problem must be at least as hard as Collatz!

Collatz and the halting problem: Another kind of reduction If we can solve the halting problem for all programs, then we can prove the Collatz conjecture. Contrapositive: If the Collatz conjecture is unproven, then the halting problem must also be unsolved. Contrapositive: If the Collatz conjecture is too hard to solve, then the halting problem must also be too hard to solve.

def halt(prog, input): ... # Assume: logic to detect halting The halting problem def halt(prog, input): ... # Assume: logic to detect halting ... return True else: return False

The halting problem: an informal proof First assume we have a program which will determine if a program P halts on input I, or not. We assume this exists and treat it as a black box. Program P halt() predicate True / False P’s input I

The halting problem: an informal proof First assume we have a program which will determine if a program P halts on input I, or not. E.g., halt() predicate collatz(n) True 9

The halting problem: an informal proof def P(prog): if halt(prog,prog): while True: pass P(P) # What happens here?

The halting problem: an informal proof Paradox! P halts if it doesn’t halt and doesn’t halt if it does! def P(prog): if halt(prog,prog): while True: pass P(P) # What happens here? Thus P cannot exist, and neither can halt.

any output to STDOUT on a given input I, or not. The printing problem Now assume we have a program which will determine if a program P prints any output to STDOUT on a given input I, or not. E.g., def f(n): if n == 3: print “hello” prints() predicate True 3

Try an example. Design a paradoxical program P which shows that decision procedure prints cannot exist.

def P(prog): if not prints(prog,prog): print “Oh, but it does though…” Try an example. Design a paradoxical program P which shows that decision procedure prints cannot exist. def P(prog): if not prints(prog,prog): print “Oh, but it does though…” P(P) # Paradox!

Reductions for decision procedures If we can reduce solving decision procedure d1 to one of solving d0, then d1’s undecidability implies d0’s. d1() predicate Do some preprocessing to turn an input to d1 to an input for d0 such that the decision for d0 is guaranteed to also be correct for d1. d0() predicate

A model of universal computation. Turing machines A model of universal computation.

deterministic finite control Informally, a Turing machine (TM) pairs a deterministic finite automaton (a set of states, alphabet, transition edges, a start state, final states) with an unbounded bi-directional tape of memory. There is no distinguished input source; the input starts out loaded into memory. deterministic finite control Edges of the TM’s finite control can read from the tape at the current position, write to the tape at the current position, and move the tape head (current position on the tape) left or right. ? q0 ? q1 ? ? E A C A A

Formally, each TM is a 7-tuple: A set of final states Formally, each TM is a 7-tuple: (𝑄,Σ,Γ,𝛿, 𝑞 0 ,𝐵,𝐹) A finite set of states {q…} A designated blank symbol. A finite set of tape symbols A finite input alphabet A deterministic transition function A designated starting state

Formally, each TM is a 7-tuple: (𝑄,Σ,Γ,𝛿, 𝑞 0 ,𝐵,𝐹) 𝛿⊆(𝑄×Γ⇀𝑄×Γ×{L,R}) 𝑞 0 ∈𝑄 𝐵∈Γ Σ⊂Γ 𝐹⊆𝑄 𝐵∉Σ

Recall delta’s signature: An example of an edge: “At state q0, if the symbol read at the current position on the tape is X: transition to q1, clear the tape at current location, and move left.” ( 𝑞 0 ,X,( 𝑞 1 ,B,L))∈𝛿 Recall delta’s signature: 𝛿⊆(𝑄×Γ⇀𝑄×Γ×{L,R})

on the memory tape, or read a Y and move left, changing Y to X.” Try an example. Give tuple(s) for the edge(s) matching the description: “At state q0, stay at q0 and either read an X and move right on the memory tape, or read a Y and move left, changing Y to X.”

on the memory tape, or read a Y and move left, changing Y to X.” Try an example. Give tuple(s) for the edge(s) matching the description: “At state q0, stay at q0 and either read an X and move right on the memory tape, or read a Y and move left, changing Y to X.” ( 𝑞 0 ,X,( 𝑞 0 ,X,R))∈𝛿 ( 𝑞 0 ,Y,( 𝑞 0 ,X,L))∈𝛿

Diagramming TMs, notational conventions E.g., If X is read, transition from q0 to q1, clearing the current memory location and moving left. ( 𝑞 1 ,B,L)=𝛿( 𝑞 0 ,X) Y/Y→ X/B← q0 q1 q2 B/X←

Transition table for TMs Y/Y→ X/B← q0 q1 q2 B/X← X Y B q0 (q1, B, L) q1 (q1, Y, R) (q2, X, L) q2 𝛿

Γ ∗ 𝑄 Γ ∗ 𝑞 0 𝑤 0 … 𝑤 𝑘 ( 𝑤 𝑖 ∈Σ) Evaluating TMs ⟨𝑄,Σ,Γ,𝛿, 𝑞 0 ,𝐵,𝐹⟩ Given a TM called M, , we can define a step- wise evaluation relation ( ) and its transitive/reflexive closure ( ). ⟨𝑄,Σ,Γ,𝛿, 𝑞 0 ,𝐵,𝐹⟩ ⊢ ⊢ ∗ 𝑥 ⊢ ∗ 𝑥 𝑥⊢𝑦 ⊢ ∗ 𝑧⟹𝑥 ⊢ ∗ 𝑧 Evaluation can be described in terms of sequences (inst. descriptions) in Γ ∗ 𝑄 Γ ∗ …starting with: 𝑞 0 𝑤 0 … 𝑤 𝑘 ( 𝑤 𝑖 ∈Σ) The input is not read separately but is loaded onto the memory tape up front, the read/write head positioned on the first input token.

𝑋 0 … 𝑋 𝑖−1 𝑞𝑋 𝑋 𝑖+1 … 𝑋 𝑘 ⊢ 𝑋 0 … 𝑋 𝑖−2 𝑝 𝑋 𝑖−1 𝑌… 𝑋 𝑘 Evaluating TMs Iff P contains an edge , then: (𝑝,𝑌,𝐿)=𝛿(𝑞,𝑋) def 𝑋 0 … 𝑋 𝑖−1 𝑞𝑋 𝑋 𝑖+1 … 𝑋 𝑘 ⊢ 𝑋 0 … 𝑋 𝑖−2 𝑝 𝑋 𝑖−1 𝑌… 𝑋 𝑘 and moving left (or right) off the end of previously-utilized memory uncovers more implicit B symbols which can then be overwritten: def 𝑞𝑋 𝑋 1 … 𝑋 𝑖−1 … 𝑋 𝑘 ⊢𝑝𝐵𝑌 𝑋 1 … 𝑋 𝑖−1 … 𝑋 𝑘

Try an example. Let’s design a TM for the language: 𝐿={xy,xx}

Try an example. Let’s design a TM for the language: 𝐿={xy,xx} x/B← x/B→ y/B← q0 q1 q2

Try an example. Show the evaluation of this TM on string xy. x/B← x/B→ y/B← q0 q1 q2

Accepts upon reaching state q2 Try an example. Show the evaluation of this TM on string xy. x/B← x/B→ y/B← q0 q1 q2 𝑞 0 xy⊢B 𝑞 1 y ⊢ 𝑞 2 BB Accepts upon reaching state q2

Try an example. Let’s design a TM for the language: 𝐿={ x 𝑛 y|𝑛∈ℕ}

𝐿={ x 𝑛 y|𝑛∈ℕ} y/y→ B/B→ x/x→ a/a→ a Try an example. Let’s design a TM for the language: 𝐿={ x 𝑛 y|𝑛∈ℕ} y/y→ B/B→ q0 q1 q2 x/x→ Note that if we replace edges in a DFA with , and add a check for a blank at the end, we get a TM! a/a→ a

Try an example. Let’s design a TM for the language: 𝐿=ℒ( 10 ∗ | 01 ∗ )

Try an example. Let’s design a TM for the language: 𝐿=ℒ( 10 ∗ | 01 ∗ ) 1/1→ q1, 0 B/B→ 0/0→ q0, B q1, B q1, 1 B/B→ 1/1→ 0/0→

Tricks for programming TMs Just as we saw with DFAs, the finite state component can also encode arbitrary information (that is perhaps inconvenient to store on the tape) so long as this information is drawn from a finite domain; e.g., 𝑄={ 𝑞 0 , 𝑞 1 }×{0,1,B} A finite domain of additional info. A TM with multiple tracks can be encoded by using a tape alphabet that is the product of multiple tape alphabets (although often the same). E.g., a tape with three synchronized tracks on the left; or two, on the right. Γ 2 Γ= Γ 1 × Γ 2 × Γ 3 Alphabet for a machine with two tracks over the same tape tokens. The alphabet for track 1, track 2, track 3.

Try an example. Let’s design a TM for the language: 𝐿={𝑥 0 𝑛 𝑥|𝑛∈ℕ∧𝑥∈{0,1}}

Try an example. Let’s design a TM for the language: 𝐿={𝑥 0 𝑛 𝑥|𝑛∈ℕ∧𝑥∈{0,1}} 0/0→ B/B← q1, 0 q2, 0 0/1→ 0/0→ q0 q3, 1 q1, 1 q2, 1 1/1→ B/B← 1/1→ 0/0→

Try an example. Let’s design a TM for the language: 𝐿={ x 𝑛 y 𝑛 |𝑛∈ℕ}

Try an example. Let’s design a TM for the language: 𝐿={ x 𝑛 y 𝑛 |𝑛∈ℕ} x/X→ q0 B/B→

Try an example. Let’s design a TM for the language: 𝐿={ x 𝑛 y 𝑛 |𝑛∈ℕ} x/X→ y/Y← q0 q1 x/x→ B/B→

Try an example. Let’s design a TM for the language: 𝐿={ x 𝑛 y 𝑛 |𝑛∈ℕ} x/x← Y/Y← x/X→ y/Y← q0 q1 q2 x/x→ B/B→

Try an example. Let’s design a TM for the language: 𝐿={ x 𝑛 y 𝑛 |𝑛∈ℕ} x/x← Y/Y← x/X→ y/Y← q0 q1 q2 x/x→ B/B→ X/X→

Try an example. Let’s design a TM for the language: 𝐿={ x 𝑛 y 𝑛 |𝑛∈ℕ} x/x← Y/Y← x/X→ y/Y← q0 q1 q2 x/x→ B/B→ Y/Y→ X/X→

Try an example. Let’s design a TM for the language: 𝐿={ x 𝑛 y 𝑛 |𝑛∈ℕ} x/x← Y/Y← x/X→ y/Y← q0 q1 q2 x/x→ B/B→ Y/Y→ Y/Y→ X/X→ q3 q4 B/B→ Y/Y→

Try an example. Show the evaluation of this TM on input xxyy. q0 q1 q2 x/x← Y/Y← B/B→ x/x→ Y/Y→ Y/Y→ q3 X/X→ q4 B/B→ Y/Y→

Try an example. Show the evaluation of this TM on input xxyy. 𝑞 0 xxyy⊢X 𝑞 1 xyy⊢Xx 𝑞 1 yy⊢X 𝑞 2 xYy⊢ 𝑞 2 XxYy ⊢X 𝑞 0 xYy⊢XX 𝑞 1 Yy⊢XXY 𝑞 1 y⊢XX 𝑞 2 YY ⊢X 𝑞 2 XYY⊢XX 𝑞 0 YY⊢XXY 𝑞 3 Y⊢XXYY 𝑞 3 B ⊢XXYYB 𝑞 4 B x/X→ y/Y← q0 q1 q2 x/x← Y/Y← B/B→ x/x→ Y/Y→ Y/Y→ q3 X/X→ q4 B/B→ Y/Y→

Tricks for programming TMs A TM with multiple (synchronized) tracks can be encoded by using a tape alphabet that is the product of multiple tape alphabets (perhaps the same). Finite control ? Γ={A,B,C,D,E} q0 q1 ? ×{T,F} ? ? E A C A A F F F T F

Tricks for programming TMs A TM with multiple (non-synchronized) tapes can be encoded by using multiple tracks where each tape is simulated by two tracks: one for its data, and one to store a mark saving the read/write head location on that tape. Finite control ? Γ={A,B,C,D,E} q0 q1 ? ×{∗,B} ? ? A A D D * * E E A A C C A A A A * *

Decidable, recognizable languages with TMs A language L (or decision problem) is decidable (also, recursive) iff there exists a TM that will always eventually halt, on any input, yielding true or false to correctly indicate the input string’s existence in language L. A language L (or decision problem) is recognizable (also, recursively enumerable or RE) iff there exists a TM that, on any input, will always eventually accept when the input is in the language L, but will either eventually reject or will non-terminate (loop forever) when the input is not in L. If L’s complement is RE, L is called co-recognizable. If a language is both recognizable and co-recognizable, then it is also decidable—i.e., both positive and negative examples can be recognized finitely.

Decidable, recognizable languages with TMs Any decision problem may be encoded formally as a language; but non-decision problems can as-well if encoded as a decision problem over pairs of (Input, Output). A problem/language is decidable if there exists a TM decider; recognizable if there is a TM recognizer; etc. Decidable : TM correctly decides and always halts. Recognizable (recursively enumerable, RE) : TM correctly decides but is only guaranteed to halt on true-yielding inputs. Co-recognizable : TM correctly decides acceptance, but is only guaranteed to halt on false-yielding inputs.

Decidable, recognizable languages with TMs Decidable problems P-TIME Recognizable / RE problems Undecidable problems / languages

A textual-encoding for TMs First, observe that TMs are amenable to a binary encoding: States can be encoded as positive ints: q1, q2, q3, … Likewise, the tape alphabet: X1, X2, X3, … Left and right directions as: D1, D2 Each edge is encoded: 0i10j10k10l10m Note that there are no sequential 1s within an edge. A TM is encoded as a sequence of its edges (in any order) separated by substring 11. A given TM can have many valid encodings. 𝛿( 𝑞 𝑖 , 𝑋 𝑗 )=( 𝑞 𝑘 , 𝑋 𝑙 , 𝐷 𝑚 )

𝐿 𝐷 An undecidable, unrecognizable language M0 M1 M2 M3 M4 Mi Enumerate all binary strings. M0 M1 The characteristic vector for Mi M2 M3 M4 . . . Mi 1 1 . . .

𝐿 𝐷 An undecidable, unrecognizable language M0 M1 M2 M3 M4 Mi Enumerate all binary strings. Note: Not all these machines need be valid; invalid TM encodings denote the empty lang. M0 M1 The characteristic vector for Mi M2 M3 M4 M1 is accepted by Mi . . . Mi 1 1 . . .

An undecidable, unrecognizable language 𝐿 𝐷 Language LD has a characteristic vector that is the complement of this diagonal. This means LD is the language of all binary strings encoding TMs that do not accept when given their own binary-string encodings. By diagonalization, LD is not RE. M0 M1 M2 1 1 M3 1 1 1 M4 1 1 1 . . . Mi 1 1 . . .

A recognizable but undecidable language 𝐿 𝑈 Language LU, called the universal language, is the language of all strings M111w such that M is an encoding of a TM that accepts on input string w. First, we observe it is recognizable because we can construct a TM M that simulates a binary-encoded TM on an input step-by-step—if the encoded TM accepts, M accepts. M could use one tape for input, one for tracking finite state, one for the simulated TM’s tape, etc. Second, we observe LU cannot be decidable because LD can be reduced its complement. If LD is not decidable, LU is not.

Reductions for decision procedures If we can reduce solving decision procedure d1 to one of solving d0, then d1’s undecidability implies d0’s. d1() predicate Do some preprocessing to turn an input to d1 into an input for d0 such that the decision for d0 is guaranteed to also be correct for d1. d0() predicate

𝐿 𝑈 A recognizable but undecidable language LD predicate Copy the input w to produce M111M. M LU predicate M111M

The language of halting TMs: 𝐿 𝐻 Assume an LH that accepts on input M111w iff machine M halts on input w. LU predicate Construct a new TM M’ that goes into an infinite loop whenever M would become stuck. We just add missing edges to an explicit stuck state! M, w LH predicate M’, w

Try the reverse reduction: can LH be reduced to LU? LH predicate ? M, w LU predicate ?

Try the reverse reduction: can LH be reduced to LU? LH predicate Construct a new TM M’ that accepts whenever M would get stuck (and also accepts if M accepts)! M, w LU predicate M’, M

Intractable problems (more in 650/750!) Infeasible or impracticable problems with no solution that we can realistically compute.

Intractable problems The main class of tractable problems is P (P-TIME): the set of problems we can solve in polynomial-time w.r.t. input size. A time-bounded TM is a TM that can solve a problem in a given complexity class (e.g., P) but not others. EXPTIME problems (problems that are time-bounded by an exponential function in the input size) are considered to be intractable for any sufficiently large input. Problems in the class NP are verifiable in polynomial time, and therefore solvable in Nondeterministic-Polynomial-time. Whether class NP is distinct from P is a major open problem.

Complexity classes for intractable problems NP-complete P NP EXPTIME, etc

Intractable problems (NP-Hard, NP-complete) NP-Hard problems are those at least as hard as problems in NP. We can prove this using a reduction from a known NP-Hard problem. If an NP-Hard problem is also in P, then all NP must be! NP-Complete problems are those that are NP and NP-Hard. Classic NP-complete problems include SAT and TSP. SAT is the problem of finding an assignment to variables which shows that a given boolean formula (using and, or, not) is satisfiable. Traveling salesman problem (TSP) asks for a hamiltonian circuit (a circuit through each node of a weighted graph once) that minimizes distance (or is below a given threshold).

Polynomial-time reductions If we can reduce solving decision procedure d1 to one of solving d0 in P-TIME, then if d0 is polynomial time, so is d1. Put another way: if d1 is unsolvable in P-TIME, then d0 also cannot be! d1() predicate Some reduction that must be polynomial-time. d0() predicate

means that P0 reduces to P1. Chains of reductions are used to prove NP-Hardness CSAT Circuit-SAT SAT 3-SAT P0 P1 means that P0 reduces to P1.

Final announcements Final Exam: Thursday Dec 13, 10:45am — 1:15pm There will be a modest curve for final letter grades. However, if your current grade is below the mid-60s %, then you will probably need to improve your grade to pass. However, the final is worth a full 20% of your grade and there will be opportunities to earn over 100% as usual. A set of new practice questions will be posted, but the final is comprehensive (roughly 2/3rds will be midterm review) so you should use all old problem sets and midterms to study! Jordan’s final office hours on Tuesday will be a practice session.

Final announcements 401: λ-calculus, interpreters/mini-compilers, formal semantics, new paradigms (functional programming, logic programming)! IDEA survey: please fill out https://uab.campuslabs.com/courseeval This is primarily how the department gets feedback on courses and instructors. Let them know how it went. I’ll be teaching 350 regularly each fall. Do you have ideas for improving future classes? Please let me know! Go ahead and fill out your survey now, in class. TM Videos: Paul Rendell’s TM coded in Conway’s game of life.