1. Fighting Knowledge Acquisition Bottleneck
ECAI 2008
Fighting Knowledge Acquisition Bottleneck
with
Argument Based Machine Learning
Martin Mozina, Matej Guid, Jana Krivec, Aleksander Sadikov and Ivan Bratko
Faculty of Computer and Information Science
University of Ljubljana, Slovenia

2. Motivation for Knowledge Acquisition with Argument Based Machine Learning
Knowledge Acquisition is a major bottleneck in building knowledge bases.
domain experts find it hard to articulate their knowledge
Machine Learning is a potential solution, but has weaknesses
Machine Learning & Knowledge Acquisition
Problem: Models are not comprehensible to domain experts
mostly statistical learning (not symbolic)
inducing spurious concepts (e.g. overfitting)
Combination of domain expert and machine learning would yield best results
learn symbolic models
exploit experts’ knowledge in learning

3. Combining Machine Learning and Expert Knowledge
Expert provides background knowledge for ML
Expert validates and revises induced theory
Iterative procedure: Experts and ML improve the model in turns
IF ... THEN ...
...

4. Combining Machine Learning and Expert Knowledge
Expert provides background knowledge for ML
Expert validates and revises induced theory
Iterative procedure: Experts and ML improve the model in turns
IF ... THEN ...
...

5. Combining Machine Learning and Expert Knowledge
Expert provides background knowledge for ML
Expert validates and revises induced theory
Iterative procedure: Experts and ML improve the model in turns
IF ... THEN ...
...
ABML

6. Definition of Argument Based Machine Learning
Learning with background knowledge:
INPUT: learning examples E, background knowledge BK
OUTPUT: theory T, T and BK explain all ei from E
Argument Based Machine Learning:
INPUT: learning examples E, arguments ai given to ei (from E)
OUTPUT: theory T, T explains ei with arguments ai
BK,T ei T ei ai

7. Argument Based Rule Learning
Name
RegularJob
Rich
AccountStatus
HairColor
CreditApproved
Mr. Bond No
Yes
Negative
Blond
Mr. Grey
Positive
Grey
Miss White
Miss Silver
Mrs. Brown
Brown
Name
RegularJob
Rich
AccountStatus
HairColor
CreditApproved
Mr. Bond No
Yes
Negative
Blond
Mr. Grey
Positive
Grey
Miss White
Miss Silver
Mrs. Brown
Brown
Name
RegularJob
Rich
AccountStatus
HairColor
CreditApproved
Mr. Bond No
Yes
Negative
Blond
Mr. Grey
Positive
Grey
Miss White
Miss Silver
Mrs. Brown
Brown
Name
RegularJob
Rich
AccountStatus
HairColor
CreditApproved
Mr. Bond No
Yes
Negative
Blond
Mr. Grey
Positive
Grey
Miss White
Miss Silver
Mrs. Brown
Brown
Classic rule learning: IF HairColor = Blond THEN CreditApproved = YES
Possible argument: Miss White received credit (CreditApproved=YES) because she has a regular job (RegularJob=YES).
AB rule learning (possible rule): IF RegularJob=YES AND AccountStatus = Positive THEN CreditApproved = YES

8. Formal definition of Argumented Examples
Argumented Example (A, C, Arguments):
A; attribute-value vector [e.g. RegularJob=YES,Rich=NO, ...]
C; class value [e.g. CreditApproved=YES]
Arguments; a set of arguments Arg1, ..., Argn for this example
Argument Argi :
Positive argument: C because Reasons
Negative Argument: C despite Reasons
Reasons: a conjunction of reasons r1, ..., rm

9. ABCN2 ABCN2 = extension of CN2 rule learning algorithm
(Clark,Niblett 1991)
Extensions:
Argument Based covering:
All conditions in R are true for E
R is consistent with at least one positive argument of E.
R is not consistent with any negative argument of E.
Evaluation: Extreme Value Correction (Mozina et al. 2006)
Probabilistic covering (required for Extreme Value Correction)

10. Interactions between expert and ABML
learn data set
Learn a hypothesis with ABML.
Find the most critical example. (if none found, stop procedure)
Expert explains the example.
Argument is added to the example.
Return to step 1.
Argument
ABML
critical example
What if expert’s explanation
is not good enough?

11. Interactions between expert and ABML
Expert explains example.
Add argument to example
Discover counter examples (if none, then stop).
Expert improves the argument for example.
Return to step 3.
Learn a hypothesis with ABML.
Find the most critical example. (if none found, stop procedure)
Expert explains the example.
Argument is added to the example.
Return to step 1.
What if expert’s explanation
is not good enough?

12. Knowledge Acquisition of Chess Concepts used in a Chess Tutoring Application
Case Study: Bad Bishop

13. The Concept of the Bad Bishop
Chess experts in general understand the concept of bad bishop.
Precise formalisation of this concept is difficult.
Traditional definition (John Watson, Secrets of Modern Chess Strategy, 1999)
A bishop that is on the same colour of squares as its own pawns is bad:
its mobility is restricted by its own pawns,
it does not defend the squares in front of these pawns.
Moreover, centralisation of these pawns is the main factor in deciding whether the bishop is bad or not.

14. Data set Data set: 200 middlegame positions from real chess games
Chess experts’ evaluation of bishops:
bad: 78 bishops
not bad: 122 bishops
wGM Jana Krivec
GM Garry Kasparov
FM Matej Guid
We randomly selected:
100 positions for learning
100 positions for testing
CRAFTY’s positional feature values served as attribute values for learning.

15. Standard Machine Learning Methods' Performance with CRAFTY's features only
Machine learning methods’ performance on initial dataset
Method CA
Brier score
AUC
Decision trees (C4.5)
73%
0,49
0,74
Logistic regression
70%
0,43
0,84
Rule learning (CN2)
72%
0,39
0,80
The results were obtained on test data set.
The results obtained with CRAFTY’s positional features only are too inaccurate for commenting purposes…
additional information for describing bad bishops is necessary.

16. First Critical Example
Rules obtained by ABML method ABCN2 failed to classify this example as "not bad"
The following question was given to the experts:
“Why is the black bishop not bad?“
The experts used their domain knowledge:
“The black bishop is not bad, since its mobility is not seriously restricted by the pawns of both players.”

17. Introducing new attributes into the domain and adding arguments to an example
Experts’ explanation could not be described with current domain attributes.
A new attribute, IMPROVED_BISHOP_MOBILITY, was included into the domain:
the number of squares accessible to the bishop, taking into account only own and opponent’s pawn structure
The argument
“BISHOP=“not bad” because IMPROVED_BISHOP_MOBILITY is high“
was added to the example.

18. Counter example
Method failed to explain critical example with given argument.
Counter example was presented to experts:
Critical example: “not bad”, IMPROVED_BISHOP_MOBILITY is high.
Counter example: “bad”, although IMPROVED_BISHOP_MOBILITY is high.
"Why is the “red” bishop bad, comparing to the “green” one?"
Experts’ explanation: “There are many pawns on the same colour of squares as the black bishop, and some of these pawns occupy the central squares.”

19. Improving Arguments with Counter Examples
Attribute BAD_PAWNS was included into the domain.
This attribute evaluates pawns that are on the colour of the square of the bishop ("bad" pawns in this sense).
The argument given to the critical example was extended to “BISHOP=“not bad” because IMPROVED_BISHOP_MOBILITY is high and BAD_PAWNS is low”
With this argument the method could not find any counter examples anymore.
New rule:
if IMPROVED_BISHOP_MOBILITY ≥ 4
and BAD_PAWNS ≤ 32
then BISHOP= “not bad” class distribution [0,39]

20. Assesing “bad” pawns
The experts designed a look-up table (left) with predefined values for the pawns that are on the color of the square of the bishop in order to assign weights to such pawns.
BAD_PAWNS_AHEAD = 16 + 24 + 2
= 42

21. After the Final Iteration...
The whole process consisted of 8 iterations.
7 arguments were attached to automatically selected critical examples
5 new attributes were included into the domain
Attribute
Description
BAD_PAWNS
pawns on the color of the square of the bishop - weighted according to their squares (bad pawns) 
BAD_PAWNS_AHEAD 
bad pawns ahead of the bishop
BAD_PAWNS _BLOCK_BISHOP_DIAGONAL
bad pawns that block the bishop's (front) diagonals 
BLOCKED_BAD_PAWNS 
bad pawns blocked by opponent's pawns or pieces 
IMPROVED_BISHOP_MOBILITY 
number of squares accessible to the bishop taking into account only pawns of both opponents 

22. Classification Accuracy Through Iterations
Results on the final dataset
Method CA
Brier score
AUC
Decision trees (C4.5)
89%
0,21
0,86
Logistic regression
88%
0,19
0,96
Rule learning (CN2)
0,94
ABCN2
95%
0,11
0,97

23. Classification Accuracy Through Iterations
The accuracies of all methods improved by adding new attributes.
ABCN2 (which also used the arguments) outperformed all others.
Arguments suggested useful attributes AND lead to even more accurate models.

24. Advantages of ABML for Knowledge Acquisition
explain single example
easier for experts to articulate knowledge
more knowledge from experts
critical examples
expert provide only relevant knowledge
time of experts' involvent is decreased

25. Advantages of ABML for Knowledge Acquisition
counter examples
detect deficiencies in expert's explanations
even more knowledge from experts
arguments constrain learning
hypotheses are consistent with expert knowledge
hypotheses comprehensible to expert
more accurate hypotheses

26. Argument Based Machine Learning better knowledge acquisition
Conclusions
ABML-based Knowledge Acquisition process provides:
more knowledge from experts
time of experts' involvent is decreased
hypotheses comprehensible to expert
more accurate hypotheses
Argument Based Machine Learning
enables
better knowledge acquisition