1. Minimax Pathology Mitja Luštrek 1, Ivan Bratko 2 and Matjaž Gams 1
1 Jožef Stefan Institute, Department of Intelligent Systems
2 University of Ljubljana, Faculty of Computer and Information Science

2. Plan of the talk What is the minimax pathology
Plan of the talk
What is the minimax pathology
Past work on the pathology
A real-valued minimax model
Why is minimax not pathological
Why is minimax beneficial
Mitja Luštrek

3. What is the minimax pathology Past work on the pathology
What is the minimax pathology
Past work on the pathology
A real-valued minimax model
Why is minimax not pathological
Why is minimax beneficial
Mitja Luštrek

4. What is the minimax pathology
What is the minimax pathology
Conventional wisdom:
the deeper one searches a game tree, the better he plays;
no shortage of practical confirmation.
Theoretical analyses:
minimaxing amplifies the error of the heuristic evaluation function;
therefore the deeper one searches, the worse he plays;
Pathology!
Mitja Luštrek

5. The pathology illustrated
The pathology illustrated
Current position
Game tree
Final values
(true)
Mitja Luštrek

6. The pathology illustrated
The pathology illustrated
Current position
Static heuristic
values (with error)
Final values
(true)
Mitja Luštrek

7. The pathology illustrated
The pathology illustrated
Current position
Backed-up heuristic
values (should be more
trustworthy, but have
larger error instead!)
Minimax
Static heuristic
values (with error)
Final values
(true)
Mitja Luštrek

8. The pathology illustrated
The pathology illustrated
Current position
Static heuristic values
(with smaller error)
Final values
(true)
Mitja Luštrek

9. What is the minimax pathology Past work on the pathology
What is the minimax pathology
Past work on the pathology
A real-valued minimax model
Why is minimax not pathological
Why is minimax beneficial
Mitja Luštrek

10. The discovery First discovered by Nau [1979].
The discovery
First discovered by Nau [1979].
A year later discovered independently by Beal [1980].
Beal’s minimax model:
uniform branching factor;
position values are losses or wins;
the proportion of losses for the side to move is constant;
position values within a level are independent of each other;
the error is the probability of mistaking a loss for a win or vice versa and is independent of the level of a position.
None of the assumptions look terribly unrealistic, yet the pathology is there.
Mitja Luštrek

11. Attempts at an explanation
Attempts at an explanation
Researchers tried to find a flaw in Beal’s model by attacking its assumptions.
Uniform branching factor:
geometrically distributed branching factor prevents the pathology [Michon, 1983];
in chess endgames asymmetrical branching factor causes the pathology [Sadikov, 2005].
Node values are losses or wins:
multiple values do not help [Bratko & Gams, 1982; Pearl, 1983];
multiple values used in a simple game -> pathological [Nau, 1982, 1983];
multiple/real values used to construct a realistic model -> not pathological [Scheucher & Kaindl, 1998; Luštrek, 2004].
Mitja Luštrek

12. Attempts at an explanation
Attempts at an explanation
The proportion of losses for the side to move is constant:
in models where it is applicable, it was agreed to be necessary [Beal, 1982; Bratko & Gams, 1982; Nau, 1982, 1983].
Node values within a level are independent of each other:
nearby positions are similar and thus have similar values;
most researchers agreed that this is the answer or at least a part of it [Beal, 1982; Bratko & Gams, 1982; Pearl, 1983; Nau, 1982, 1983; Schrüfer, 1986; Scheucher & Kaindl, 1998; Luštrek, 2004].
Mitja Luštrek

13. Attempts at an explanation
Attempts at an explanation
The error is independent of the level of a position:
varying error cannot account for the absence of the pathology [Pearl, 1983];
used varying error in a game and it did not help [Nau, 1982, 1983];
varying error is a part of the answer (with the other part being node-value dependence) [Scheucher & Kaindl, 1998].
Despite some disagreement, node-value dependence seems to be the most widely supported explanation.
But is it really necessary? Is there no simpler, more fundamental explanation? We believe there is!
Mitja Luštrek

14. What is the minimax pathology Past work on the pathology
What is the minimax pathology
Past work on the pathology
A real-valued minimax model
Why is minimax not pathological
Why is minimax beneficial
Mitja Luštrek

15. Why multiple/real values?
Why multiple/real values?
Necessary in games where the final outcome is multivalued (Othello, tarok).
Used by humans and game-playing programs.
Seem unnecessary in games where the outcome is a loss, a win or perhaps a draw (chess, checkers).
But:
in a losing position against a fallible and unknown opponent, the outcome is uncertain;
in a winning position, a perfect two-valued evaluation function will not lose, but it may never win, either.
Multiple values are required to model uncertainty and to maintain a direction of play towards an eventual win.
Mitja Luštrek

16. Why multiple/real values?
Why multiple/real values?
If only the positions along the quickest / most probable path to victory were evaluated as won, even two values would enable optimal play.
Strange: won position would be evaluated as lost.
Not quite what the researchers using two values had in mind.
But still ... ?
Mitja Luštrek

17. A real-valued minimax model
A real-valued minimax model
Aims to be a real-valued version of Beal’s model:
uniform branching factor;
position values are real numbers;
if the real values are converted to losses and wins, the proportion of losses for the side to move is constant;
position values within a level are independent of each other;
the error is normally distributed noise and is independent of the level of a position.
The crucial difference is the assumption 5.
Mitja Luštrek

18. Assumption 5 - + Two-value error: Real-value error: Loss Win 0.31 0.74
Assumption 5
Two-value error:
Real-value error: - +
Loss
Win
0.31
0.74
Mitja Luštrek

19. Assumption 5 Beal’s assumption 5:
Assumption 5
Beal’s assumption 5:
Static P (loss ↔ win) constant with the depth of search.
Our assumption 5:
The magnitude of static real-value noise constant with the depth of search.
P (loss ↔ win)
Real-value noise
Depth
Depth
Note: static = applied at the lowest level of search.
Mitja Luštrek

20.
Building of a game tree
Mitja Luštrek

21. Building of a game tree True values distributed uniformly in [0, 1]
Building of a game tree
True values distributed uniformly in [0, 1]
Mitja Luštrek

22.
Building of a game tree
True values backed up
Mitja Luštrek

23.
Building of a game tree
True values backed up
Mitja Luštrek

24.
Building of a game tree
True values backed up
Mitja Luštrek

25.
Building of a game tree
True values backed up
Mitja Luštrek

26.
Building of a game tree
Search to this depth
Mitja Luštrek

27. Building of a game tree Heuristic values = true values +
Building of a game tree
Heuristic values = true values +
normally distributed noise
Mitja Luštrek

28. Building of a game tree Heuristic values backed up 2005-12-08
Mitja Luštrek

29. Building of a game tree Heuristic values backed up 2005-12-08
Mitja Luštrek

30. Building of a game tree Heuristic values backed up 2005-12-08
Mitja Luštrek

31. What we do with our model
What we do with our model
Monte Carlo experiments:
generate 10,000 sets of true values;
generate 10 sets of heuristic values per set of true values per depth of search.
Measure the error at the root:
real-value error = the average difference between the true value and the heuristic value;
two-value error = the frequency of mistaking a loss for a win or vice versa.
Compare the error at the root when searching to different depths.
Mitja Luštrek

32. Conversion of real values to losses and wins
Conversion of real values to losses and wins
To measure two-value error, real values must be converted to losses and wins.
Value above a threshold means win, below the threshold loss.
At the leaves:
the proportion of losses for the side to move = cb (because it must be the same at all levels);
real values distributed uniformly in [0, 1];
therefore threshold = cb.
At higher levels:
minimaxing on real values is equivalent to minimaxing on two values;
therefore also threshold = cb.
Mitja Luštrek

33. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Mitja Luštrek

34. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Minimaxing
Mitja Luštrek

35. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Minimaxing
Mitja Luštrek

36. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Apply threshold
Mitja Luštrek

37. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Mitja Luštrek

38. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Mitja Luštrek

39. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Apply
threshold
Mitja Luštrek

40. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Minimaxing
Mitja Luštrek

41. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Minimaxing
Mitja Luštrek

42. Conversion of real values to losses and wins
Conversion of real values to losses and wins
Real values
Two values
Mitja Luštrek

43. What is the minimax pathology Past work on the pathology
What is the minimax pathology
Past work on the pathology
A real-valued minimax model
Why is minimax not pathological
Why is minimax beneficial
Mitja Luštrek

44. Error at the root / constant static real-value error
Error at the root / constant static real-value error
Plotted: real-value and two-value error at the root.
Static real-value error: normally distributed noise with standard deviation 0.1.
Mitja Luštrek

45. Static two-value error / constant static real-value error
Static two-value error / constant static real-value error
Plotted: static two-value error.
Static real-value error: normally distributed noise with standard deviation 0.1.
Mitja Luštrek

46. Static real-value error / constant static two-value error
Static real-value error / constant static two-value error
Plotted: static real-value error.
Static two-value error: 0.1.
Mitja Luštrek

47. Error at the root / constant static two-value error
Error at the root / constant static two-value error
Plotted: two-value error at the root in our real-value model and in Beal’s model.
Two-value error at the lowest level of search: 0.1.
After a small tweak of Beal’s model, we get a perfect match.
Mitja Luštrek

48. Conclusions from the graphs
Conclusions from the graphs
Static real-value is constant:
static two-value error decreases with the depth of search;
no pathology.
Static two-value error is constant:
static real-value error increases with the depth of search;
pathology.
Which static error should be constant?
Mitja Luštrek

49. Should real- or two-value static error be constant?
Should real- or two-value static error be constant?
Explained why real values are necessary.
Real-value error most naturally represents the fallibility of the heuristic evaluation function.
Game playing programs do not use two-valued evaluation functions, but if they did:
they would more often make a mistake in uncertain positions close to the threshold;
they would rarely make a mistake in certain positions far from the threshold.
Mitja Luštrek

50. Should real- or two-value static error be constant?
Should real- or two-value static error be constant?
Mitja Luštrek

51. Verification in trioomph
Verification in trioomph
Mitja Luštrek

52. Verification in trioomph
Verification in trioomph
Mitja Luštrek

53. Average difference? A simple solution, but perhaps not the best one.
Average difference?
A simple solution, but perhaps not the best one.
A table of all possible combinations of errors might give some insight.
Mitja Luštrek

54. Average difference? A simple solution, but perhaps not the best one.
Average difference?
A simple solution, but perhaps not the best one.
A table of all possible combinations of errors might give some insight.
Mitja Luštrek

55. Average difference? A simple solution, but perhaps not the best one.
Average difference?
A simple solution, but perhaps not the best one.
A table of all possible combinations of errors might give some insight.
Mitja Luštrek

56. Average difference? A simple solution, but perhaps not the best one.
Average difference?
A simple solution, but perhaps not the best one.
A table of all possible combinations of errors might give some insight.
Mitja Luštrek

57. Average difference? A simple solution, but perhaps not the best one.
Average difference?
A simple solution, but perhaps not the best one.
A table of all possible combinations of errors might give some insight.
Mitja Luštrek

58. Two-value error larger at higher levels
Two-value error larger at higher levels
Some simplifications:
branching factor = 2;
node values in [0, 1];
consider only one type of error: wins mistaken for losses;
consider two levels at a time to avoid even/odd level differences.
X ... true real value of a node F (x) = P (X < x) ... distribution function of the true real value e ... real-value error X – e ... heuristic real value t ... threshold
Two-value error: P (X > t  X – e < t) = P (t < X < t + e) = F (t + e) – F (t)
Mitja Luštrek

59. Two-value error larger at higher levels
Two-value error larger at higher levels
We need to show that two-value error at higher levels is larger than at lower levels:
Fi – 2 (t + e) – Fi – 2 (t) > Fi (t + e) – Fi (t)
the difference in F between the points t + e and t is larger at higher levels, which means that F is steeper at higher levels.
Example: uniform distribution at the leaves, depth = 10:
F10 (x) = x
F8 (x) = 4 x 2 – 4 x 3 + x 4
Mitja Luštrek

60. Two-value error larger at higher levels
Two-value error larger at higher levels
F8 (x) steeper than F10 (x) between x = a and x = b.
Mitja Luštrek

61. Two-value error larger at higher levels
Two-value error larger at higher levels
In general, two-value error at higher levels is larger than at lower levels if: F (a) = < F (t) < = F (b)
Why can we expect this condition to be true:
F (t) = the proportion of losses, which is constant;
a constant proportion of losses is achieved by each player having just enough advantage after his move so that the opponent can balance it out after his move;
when one’s advantage is too large, it gets even larger at each successive level;
therefore F (t) can be expected not to be very large or very small.
Mitja Luštrek

62. Two-value error sufficiently larger at higher levels
Two-value error sufficiently larger at higher levels
We have shown that two-value error at higher levels is larger than at lower levels.
Is it larger enough?
Baseline:
when searching to the maximum depth, we compute two-value error for all levels: pi ... the error at level i;
when searching to depth d, two-value error at depth d = pd .
Two-value error at level i is larger than the baseline when:
Mitja Luštrek

63. Two-value error sufficiently larger at higher levels
Two-value error sufficiently larger at higher levels
Pathology only when F (t + e) close to 1:
F (t) not expected to be close to 1;
therefore this means a very large error.
Mitja Luštrek

64. Minimax is not pathological
Minimax is not pathological
Real-valued evaluations are necessary for successful game-playing.
If static real-value error is constant, static two-value error is larger when searching to smaller depths.
This happens because at smaller depths, node values are closer to the threshold separating losses from wins.
The pathology is thus eliminated.
Our explanation for the lack of pathology is a necessary consequence of a real-value minimax model and requires no additional assumptions.
What happens if we use a number of discrete values instead of real values (which is more realistic)?
Mitja Luštrek

65. What is the minimax pathology Past work on the pathology
What is the minimax pathology
Past work on the pathology
A real-valued minimax model
Why is minimax not pathological
Why is minimax beneficial
Mitja Luštrek

66. Preamble Will only consider real values.
Preamble
Will only consider real values.
Must compare searches to different depths of the same tree:
the shape of the tree does not matter;
what matters is that each minimaxing step reduces the error;
therefore more steps mean smaller error.
Some simplifications:
constant difference between the values of sibling nodes;
branching factor = 2.
Mitja Luštrek

67.
One minimaxing step
Mitja Luštrek

68.
Ten minimaxing steps
Mitja Luštrek

69.
Conclusion
Theoretical analyses in the past have shown that minimax is pathological.
The explanations that followed have introduced unnecessary complications.
If the error is modeled in the way a real-valued model suggests, the pathology disappears.
Real values also lend themselves well to an explanation of why is minimax beneficial.
Mitja Luštrek

70. Thank you. Questions?