Empirical Explorations with The Logical Theory Machine: A Case Study in Heuristics by Allen Newell, J. C. Shaw, & H. A. Simon by Allen Newell, J. C. Shaw, & H. A. Simon

u The Logic Theory Machine (LT) : a system for finding proofs of theorems in elementary symbolic logic : a system for finding proofs of theorems in elementary symbolic logic : devised to understand the complex problem- solving process : devised to understand the complex problem- solving process : the first heuristic program (1956) : the first heuristic program (1956)

u The Logic Theory Machine in Operation - LT’s Task is to prove theorems in the sentential calculus ( in Principia Mathematica ) sentential calculus ( in Principia Mathematica ) - The proof is the sequence of expressions that leads from the axioms and known theorems to the desired expression e.g.) - Why are such problems difficult ? - What features of LT account for its successes and failures ? and failures ?

u Problems, Algorithms, and Heuristics w problem : if he is given some process for generating the elements of a set of possible solutions in some order, and a test for verifying whether a given element of this set is in fact a solution to his problem. : if he is given some process for generating the elements of a set of possible solutions in some order, and a test for verifying whether a given element of this set is in fact a solution to his problem. w problem solving : if the costs are not too large in relation to the time and computing power available for solution : if the costs are not too large in relation to the time and computing power available for solution

u Problems, Algorithms, and Heuristics w algorithm : the process that guarantee the solution of a given problem. ex) opening a combination safe : the process that guarantee the solution of a given problem. ex) opening a combination safe w heuristic : the process that may solve a given problem but offers no guarantees of doing so. : the process that may solve a given problem but offers no guarantees of doing so. ex) playing chess ex) playing chess

u The Problem of Proving Theorems in Logic - the set of all possible sequences of logic expression : E ( given implicitly by rules for generating) : E ( given implicitly by rules for generating) - the set of sequences that are proofs : P - the expression to be proved : X - the set of sequences that have X as their final expression : T x P :proof sequence T x : sequences ending in X E : all sequences of logic expressions Proofs of X

w The difficulty of proving theorems depends on 1. The scarcity of elements in the intersection of P and T x, relative to the number of elements in E 1. The scarcity of elements in the intersection of P and T x, relative to the number of elements in E  the cost and speed of the available generators,  the cost and speed of the available generators, the cost and speed of making tests (whether P or T x the cost and speed of making tests (whether P or T x ) that any element they produce automatically satisfies some of the conditions. 2. Whether generators can be found that guarantee that any element they produce automatically satisfies some of the conditions. 3. What heuristics can be found to guide the selection. 3. What heuristics can be found to guide the selection.

z The British Museum Algorithm : reveal the basic nature of theorem proving : reveal the basic nature of theorem proving w the algorithm constructs all possible proofs in systematic manner, checking each time (1) to eliminate duplicates, and (2) to see if the final theorem in the proof coincides with the expression to be proved w the set of n-step proofs : obtained from the set of (n-1)-step proofs by making all the permissible substitutions and replacements, detachments.

w Figure 2. Number of proofs generated by first few steps of British Museum algorithm : only use the simple substitution : only use the simple substitution and the replacements and the replacements : If specializations are included : If specializations are included 11-steps, 1000 theorems 11-steps, 1000 theorems : among the first 246 theorems, : among the first 246 theorems, only 5 theorem ( in chap. 2 of only 5 theorem ( in chap. 2 of principia), 1000 theorems, one principia), 1000 theorems, one more more w how many theorem to prove all theorems (60-odd) : a hundred million : a hundred million w how many times : hundreds of thousands of years

u The Logic Theory Machine z Methods : the major type of heuristic that LT uses : the major type of heuristic that LT uses The Substitution Method : by finding an axiom or proved theorem that can be transformed, by a series of substitutions for variables and replacements of connectives : by finding an axiom or proved theorem that can be transformed, by a series of substitutions for variables and replacements of connectives The Detachment Method : if problem B, search for an axiom or theorem “ A implies B ”, A is set up as a new subproblem : if problem B, search for an axiom or theorem “ A implies B ”, A is set up as a new subproblem

z Methods (continued) The Chaining Method : use the tansitivity of the relation of implication to create a new subproblem new subproblem - forward chaining method : “ a implies c ”, search an axiom or theorem “ a implies b ”, set up “ b implies c ” as new subproblem : “ a implies c ”, search an axiom or theorem “ a implies b ”, set up “ b implies c ” as new subproblem - backward chaining method : search “ b implies c ”, set up “ a implies b ” as subproblem : search “ b implies c ”, set up “ a implies b ” as subproblem

z Methods (continued) w no guarantee : a target axiom or a theorem can be found : a target axiom or a theorem can be found : the generated subproblem is part of the desired proof sequence, or part of any proof sequence : the generated subproblem is part of the desired proof sequence, or part of any proof sequence : the beginning of the sequence can be completed with axioms or proved theorems : the beginning of the sequence can be completed with axioms or proved theorems : the combination of four methods comprise a sufficient set of methods to prove all theorems : the combination of four methods comprise a sufficient set of methods to prove all theorems ex) “ p or not-not-not-p ” (2.13) ex) “ p or not-not-not-p ” (2.13)

z Methods (continued) w Why do the methods transform LT into an effective problem-solver ? 1. unitizing effect 1. unitizing effect : the methods organize the sequences of individual processing steps into larger units : the methods organize the sequences of individual processing steps into larger units 2. backward approach 2. backward approach : only one theorem to be proved, a number of known true theorems : only one theorem to be proved, a number of known true theorems

z The Executive Routine 1. For a theorem list, the substitution 1. For a theorem list, the substitution method is tried method is tried 2. The detachment method is tried, 2. The detachment method is tried, following the substitution, if fail, following the substitution, if fail, the subproblem is added to the the subproblem is added to the subproblem list subproblem list 3. Forward chaining, backward 3. Forward chaining, backward chaining. chaining. 4. For the subproblem list, repeat. 4. For the subproblem list, repeat.

n The Executive Routine (continued) This process continues until (1) a proof is found, (2) the time allotted for finding a proof is used up, (3) there is no more available memory space, (4) no untried problems remain on the subproblem list listExamples

n The Executive Routine (continued) not (p or q) implies not-p not (p or q) implies not-p (A implies B) implies (not-B (A implies B) implies (not-B implies not-A) (theorem 2.16) implies not-A) (theorem 2.16) subprblems [ p implies (p or q)] subprblems [ p implies (p or q)] by substitution method by substitution method A implies (A or B) A implies (A or B)

n The Matching Process w What allows the efficient search ? w the required times : BM algorithm vs LT ( 5~100:1) ex) “ p implies (q implies p) ” (158 sec : 10 sec) ex) “ p implies (q implies p) ” (158 sec : 10 sec) p implies (q or p) (axiom 1.3) p implies (q or p) (axiom 1.3) p implies (not-q or p) p implies (not-q or p) p imlies (q implies p) p imlies (q implies p)

n The Matching Process (continued) w Matching in the Substitution Methods : componentwise matching - a feedback of the results of a substitution or replacement : componentwise matching - a feedback of the results of a substitution or replacement : Of the 52 theorems of Principia, 38 proofs : Of the 52 theorems of Principia, 38 proofs substitution alone - 17 proof substitution alone - 17 proof w Matching in Detachment and Chaining : matching the subexpressions of the problem or theorems. : matching the subexpressions of the problem or theorems.

n Similarity Tests and Descriptions w Screening Process : reject any theorem for matching that has low likelihood of success w The Similarity Test : 1) the maximum numbers of levels, 2) the number of distinct variables, 3) the number of variable places w table 1.

n Similarity Tests and Descriptions (cont.) w similarity test 의 문제점 : type II error ex) p implies (p or p)  p implies (q or p) : type II error ex) p implies (p or p)  p implies (q or p) : type I error : type I error : the cost of similarity test : the cost of similarity test n Effort in LT w w measuring efforts : the total number of primitives executed w w computing effort and performance : substitution vs detach. vs chain. ( 1: 2 : 3)

n Similarity Tests and Descriptions (cont.) - precomputed, recomputed description - precomputed, recomputed description - the effort required is proportional to the number of theorems considered. - the effort required is proportional to the number of theorems considered. - the number of theorems considered is an effort measure for evaluating a heuristic - the number of theorems considered is an effort measure for evaluating a heuristic w Evaluation of the Similarity Test : the full similarity vs the modified similarity vs no similarity : the full similarity vs the modified similarity vs no similarity : when precomputed, full : no (10 to 60 perc. 3534/5206) : when precomputed, full : no (10 to 60 perc. 3534/5206) when recomputed, (26,739/22,914) when recomputed, (26,739/22,914) : modified similarity test (18,035/22,914) : modified similarity test (18,035/22,914)

u Subproblems figure 6. Distribution of LT’s proofs by effort w the limitation of LT : a “plateau” require quite different heuristics : a “plateau” require quite different heuristics z The Subproblem Tree

n The Subproblem Tree w theorem 2.17 : the proof that cost LT 89,000 primitives. (not-q implies not-p) implies (p implies q) (theorem 2.17) (not-q implies not-p) implies (p implies q) (theorem 2.17) 1. A implies not-not-A 2. p implies not-not-p 3. (A implies B) implies [(B implies C) implies (A implies C)] (theorem 2.06) 2.06) 4. (p implies not-not-p) implies [(not-not-p implies q) implies (p implies q) 5. (not-not-p implies q) implies (p implies q) (det. 4 from 3) 6. (not-A implies B) implies (not-B implies A) 7. (not-q implies not-p) implies (not-not-p implies q) 8. (not-q implies not-p) implies (p implies q) (chain 7 and 5)

n The Subproblem Tree step 3 step 4 step 3 step 4 : the difference in total effort can be attributed largely to the difference in number of subproblems generated : an algorithmic procedure to govern its generation of subproblem, an algorithm to determine the order to try

n Modification of the Logic Theory Machine 1. The unit cost of processing subproblems can be reduced. 2. LT can be modified to screen subproblems before they are put on the subproblem list. 3. Another way is to reduce selectively the number of subproblems generated, by limiting the lists for theorems available for generating subproblems. Cf) Fig. 7. A list fo 20 theorems, fig theorems Cf) Fig. 7. A list fo 20 theorems, fig theorems ex) theorem (2.48) ex) theorem (2.48) two condition: all theorems vs axiom + one theorem two condition: all theorems vs axiom + one theorem the longer list : in two steps, 51,450 primitives of effort the longer list : in two steps, 51,450 primitives of effort the shorter list : in tree step, 18,558 primitives the shorter list : in tree step, 18,558 primitives

u Conclusion - Heuristics give the program power to solve problems in reasonable computing time - The limitations of the present program of LT - Some of directions that improvement would have to take to extend it powers to problems at new to take to extend it powers to problems at new levels of difficulty levels of difficulty

u The Definitions of the Sentential Calculus - variables p, q, r, … A, B, C … - variables p, q, r, … A, B, C … - connectives not, or, implies - connectives not, or, implies - expression “not-p”, “p or q”, “p implies q” - expression “not-p”, “p or q”, “p implies q” - axiom the universally true expressions - axiom the universally true expressions - theorem - theorem the true expressions that are derived from axioms by means of various rules of inference the true expressions that are derived from axioms by means of various rules of inference - rules of inference formalization of logical operation ( e.g. deduction) - rules of inference formalization of logical operation ( e.g. deduction)

u the system of axioms ( p or p ) implies p (1.2) ( p or p ) implies p (1.2) p implies ( q or p ) (1.3) p implies ( q or p ) (1.3) ( p or q ) implies ( q or p ) (1.4) ( p or q ) implies ( q or p ) (1.4) [ p or ( q or r ) ] implies [ q or ( p or r ) ] (1.5) [ p or ( q or r ) ] implies [ q or ( p or r ) ] (1.5) ( p implies q ) implies [ ( r or p ) implies ( r or q ) ] (1.6) ( p implies q ) implies [ ( r or p ) implies ( r or q ) ] (1.6) u three rules of inference 1. Substitution : “ A” for “ p ”, “ p or q ” for “ p ” 1. Substitution : “ A” for “ p ”, “ p or q ” for “ p ” 2. Replacement : “ p implies q ” with “ not-p or q ” 2. Replacement : “ p implies q ” with “ not-p or q ” 3. Detachment : if “ A ” and “ A implies B ”, then “ B” 3. Detachment : if “ A ” and “ A implies B ”, then “ B”

u ( p implies not-p) implies not-p (theorem 2.01) 1. ( A or A ) implies A (axiom 1.2) 1. ( A or A ) implies A (axiom 1.2) 2. ( not-A or not-A ) implies not-A 2. ( not-A or not-A ) implies not-A 3. ( A implies not-A ) implies not-A 3. ( A implies not-A ) implies not-A 4. ( p implies not-p ) implies not-p 4. ( p implies not-p ) implies not-p : LT works for about 10 seconds : LT works for about 10 seconds

u not ( p or q ) implies not-p (theorem 2.45) 1. A implies (A or B ) (theorem 2.2) 1. A implies (A or B ) (theorem 2.2) 2. p implies ( p or q ) 2. p implies ( p or q ) 3. (A implies B ) implies (not-B implies not-A) (theorem 2.16) 3. (A implies B ) implies (not-B implies not-A) (theorem 2.16) 4. [ p implies ( p or q )] implies [ not (p or q) implies not-p) 4. [ p implies ( p or q )] implies [ not (p or q) implies not-p) 5. not ( p or q ) implies not-p 5. not ( p or q ) implies not-p  works for about 12 minutes and success  works for about 12 minutes and success u [ p or ( q or r ) ] implies [ ( p or q ) or r ] (theorem 2.31)  works for about 23 minutes and fails to prove