1. Introduction to ILP ILP = Inductive Logic Programming = machine learning  logic programming = learning with logic Introduced by Muggleton in 1992

2. (Machine) Learning The process by which relatively permanent changes occur in behavioral potential as a result of experience. (Anderson) Learning is constructing or modifying representations of what is being experienced. (Michalski) A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E. (Mitchell)

3. Machine Learning Techniques Decision tree learning Conceptual clustering Case-based learning Reinforcement learning Neural networks Genetic algorithms and… Inductive Logic Programming

4. Why ILP ? - Structured data Seed example of East-West trains (Michalski) What makes a train to go eastward ?

5. Why ILP ? – Structured data Mutagenicity of chemical molecules (King, Srinivasan, Muggleton, Sternberg, 1994) What makes a molecule to be mutagenic ?

6. Why ILP ? – multiple relations This is related to structured data TrainCar t1 c11 t1 c12 t1 c13 t1 c14 t2 c21 … … CarLengthShapeAxesRoof… c11shortrectangle2none… c12longrectangle3none… c13shortrectangle2peaked… c14longrectangle2none… c21shortrectangle2flat… ……………… has_carcar_properties

7. Why ILP ? – multiple relations Genealogy example: Given known relations… –father(Old,Young) and mother(Old,Young) –male(Somebody) and female(Somebody) …learn new relations –parent(X,Y) :- father(X,Y). –parent(X,Y) :- mother(X,Y). –brother(X,Y) :- male(X),father(Z,X),father(Z,Y). Most ML techniques can’t use more than 1 relation e.g.: decision trees, neural networks, …

8. Why ILP ? – logical foundation Prolog = Programming with Logic is used to represent: –Background knowledge (of the domain): facts –Examples (of the relation to be learned): facts –Theories (as a result of learning): rules Supports 2 forms of logical reasoning –Deduction –Induction

9. Prolog - definitions Variables: X, Y, Something, Somebody Terms: arthur, 1, [1,2,3] Predicates: father/2, female/1 Facts: –father(christopher,victoria). –female(victoria). Rules: –parent(X,Y) :- father(X,Y).

10. Logical reasoning: deduction From rules to facts… B  T |- E mother(penelope,victoria). mother(penelope,arthur). father(christopher,victoria). father(christopher,arthur). parent(X,Y) :- father(X,Y). parent(X,Y) :- mother(X,Y). parent(penelope,victoria). parent(penelope,arthur). parent(christopher,victoria). parent(christopher,arthur).

11. Logical reasoning: induction From facts to rules… B  E |- T mother(penelope,victoria). mother(penelope,arthur). father(christopher,victoria). father(christopher,arthur). parent(X,Y) :- father(X,Y). parent(X,Y) :- mother(X,Y). parent(penelope,victoria). parent(penelope,arthur). parent(christopher,victoria). parent(christopher,arthur).

12. Induction of a classifier or Concept Learning Most studied task in Machine Learning Given: –background knowledge B –a set of training examples E –a classification c  C for each example e Find: a theory T (or hypothesis) such that B  T |- c(e), for all e  E

13. Induction of a classifier: example Example of East-West trains B: relations has_car and car_properties ( length, roof, shape, etc.) ex.: has_car(t1,c11), shape(c11,bucket) E: the trains t1 to t10 C: east, west

14. Why ILP ? - Structured data Seed example of East-West trains (Michalski) What makes a train to go eastward ?

15. Induction of a classifier: example Example of East-West trains B: relations has_car and car_properties ( length, roof, shape, etc.) ex.: has_car(t1,c11) E: the trains t1 to t10 C: east, west Possible T: east(T) :- has_car(T,C), length(C,short), roof(C,_).

16. Induction of a classifier: example Example of mutagenicity B: relations atom and bond ex.: atom(mol23,atom1,c,195). bond(mol23,atom1,atom3,7). E: 230 molecules with known classification C: active and nonactive w.r.t. mutagenicity Possible T: active(Mol) :- atom(Mol,A,c,22), atom(Mol,B,c,10), bond(Mol,A,B,1). c22 c10

17. Learning as search Given: –Background knowledge B –Theory Description Language T –Positives examples P (class +) –Negative examples N (class -) –A covering relation covers(B,T,e) Find: a theory that covers –all positive examples (completeness) –no negative examples (consistency)

18. Learning as search Covering relation in ILP covers(B,T,e)  B  T |- e A theory is a set of rules Each rule is searched separately (efficiency) A rule must be consistent (cover no negatives), but not necessary complete Separate-and-conquer strategy –Remove from P the examples already covered

19. Space exploration Strategy? Random walk –Redundancy, incompleteness of the search Systematic according to some ordering –Better control => no redundancy, completeness –The ordering may be used to guide the search towards better rules What kind of ordering?

20. Generality ordering Rule 1 is more general than rule 2 => Rule 1 covers more examples than rule 2 –If a rule is consistent (covers no negatives) then every specialisation of it is consistent too –If a rule is complete (covers all positives) then every generalisation of it is complete too Means to prune the search space 2 kinds of moves: specialisation and generalisation Common ILP ordering: θ-subsumption

21. Generality ordering parent(X,Y):- parent(X,Y):- female(X)parent(X,Y) :- father(X,Y) parent(X,Y) :- female(X), mother(X,Y) parent(X,Y) :- female(X), father(X,Y) consistent rulespecialisation

22. Search biases “Bias refers to any criterion for choosing one generalization over another other than strict consistency with the observed training instances.” (Mitchell) Restrict the search space (efficiency) Guide the search (given domain knowledge) Different kinds of bias –Language bias –Search bias –Strategy bias

23. Choice of predicates: roof(C,flat) ? roof(C) ? flat(C) ? Types of predicates : east(T) :- roof(T), roof(C,3) Modes of predicates : east(T) :- roof(C,flat) east(T) :- has_car(T,C), roof(C,flat) Discretization of numerical values Language bias

24. Search bias The moves direction in the search space Top-down –start: the empty rule (c(X) :-.) –moves: specialisations Bottom-up –start: the bottom clause (~ c(X) :- B.) –moves: generalisations Bi-directional

25. Strategy bias Heuristic search for a best rule Hill-climbing: –Keep only one rule –efficient but can miss global maximum Beam search: –also keep k rules for back-tracking –less greedy Best-first search: –keep all rules –more costly but complete search

26. A generic ILP algorithm procedure ILP(Examples) Initialize(Rules, Examples) repeat R = Select(Rules, Examples) Rs = Refine(R, Examples) Rules = Reduce(Rules+Rs, Examples) until StoppingCriterion(Rules, Examples) return(Rules)

27. A generic ILP algorithm Initialize(Rules,Examples): initialize a set of theories as the search starting points Select(Rules,Examples): select the most promising candidate rule R Refine(R,Examples): returns the neighbours of R (using specialisation or generalisation) Reduce(Rules,Examples): discard unpromising theories (all but one in hill-climbing, none in best- first search)

29. ILPnet2 – www.cs.bris.ac.uk/~ILPnet2/ Network of Excellence in ILP in Europe 37 universities and research institutes Educational materials Publications Events (conferences, summer schools, …) Description of ILP systems Applications

30. ILP systems FOIL (Quinlan and Cameron-Jones 1993): top- down hill-climbing search Progol (Muggleton, 1995): top-down best-first search with bottom clause Golem (Muggleton and Feng 1992): bottom-up hill-climbing search LINUS (Lavrac and Dzeroski 1994): propositionalisation Aleph (~Progol), Tilde (relational decision trees), …

31. ILP applications Life sciences –mutagenecity, predicting toxicology –protein structure/folding Natural language processing –english verb past tense –document analysis and classification Engineering –finite element mesh design Environmental sciences –biodegradability of chemical compounds

32. The end A few books on ILP… J. Lloyd. Logic for learning: learning comprehensible theories from structured data. 2003.Logic for learning: learning comprehensible theories from structured data S. Dzeroski and N. Lavrac, editors. Relational Data Mining. September 2001.Relational Data Mining L. De Raedt, editor. Advances in Inductive Logic Programming. 1996.Advances in Inductive Logic Programming N. Lavrac and S. Dzeroski. Inductive Logic Programming: Techniques and Applications. 1994.Inductive Logic Programming: Techniques and Applications