1. 1 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence How to Use and Abuse MCDA and Consensus Support Methods Dave Mason Operational Research Division National Defence Headquarters Ottawa, Canada

2. 2 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Outline l Definitions and applications l Cardinal utility methods »When they are applicable, desired attributes »How to abuse »Summary: How To Use Cardinal Utility Methods l Ordinal methods »Fundamental concepts (Arrow’s Theorem, Voter’s Paradox) »How to abuse common ordinal methods »Heading towards a solution to the Consensus Ranking Problem –Solution concept: Maximization of rank correlation –Existing statistics: Kendall’s τ b, Kemeny/Snell distance –Unified theories, and a new rank correlation statistic: τ X »Summary: How to Use Ordinal Methods

3. 3 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence MCDA and Consensus Support Methods l Multi-Criteria Decision Analysis Methods »Generally, any scheme that evaluates a list of alternatives across a set of assessment criteria to determine a winner (or final ordering) –Weighted sum methods are the most common form l Consensus Support Methods »Any methodology that attempts to resolve value judgements of a group of individuals into a single collective set of judgements

4. 4 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Applicable Decision Environments l Applicable types of decision environments: »Evaluation of contract bids »Ranking and selection of personnel »Prioritization of an organization’s work program »Inevitable stream of short-fuse ‘Tiger Team’ approaches »… l Generally, involves one of following types of decisions: »1. Single Choice - selecting the winning alternative »2. Prioritization - put the list of alternatives in a rank ordering »(3. Resource Allocation – assign resources to each alternative) l Methods are applied after the conduct of, not in lieu of a comprehensive cost-effectiveness analyses

5. 5 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence A Single Overall Measure of Utility? l Key question the analyst must ask: “Is the development of a single overall measure of utility reasonable in this case?” »A single measure may defy definition »Is a potentially dangerous over-simplification »But may be a necessary evil to enable an answer to be reached l YES. Then a cardinal utility method may be useful l NO. Then an ordinal methods approach may be a useful alternative

6. 6 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence CARDINAL UTILITY METHODS l Generally the broad class of weighted sum methods are most commonly employed »Assign weights to key criteria »Score the alternatives on each criteria »Arithmetically calculate the weighted sum l Numerous approaches with enhanced mathematical complexity exist »Analytical Hierarchy Process (AHP) »…

7. 7 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence (Clear glass doors) ?? “Black Box” Method “Black Box” Method Qualitative and Quantitative Inputs Final winner or ordering Desirable Attributes: 1. Transparency of Method

8. 8 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Desirable Attributes: 2. Reasonable Fidelity l Complex enough to adequately represent the key dimensions of overall utility, »“KISS” is fine, but don’t go too far l More Sophisticated Mathematics = A Better Solution? »Not necessarily l Oversold obscure methods with gratuitous mathematical complexity = Snake Oil? »You bet!

9. 9 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence How To Abuse Weighted Sum Methods Weight: Sports Car Sedan Mini-Van SUV 3 4 5 10 8 Cost Fun Style Reliability Versatility Total 2 6 10 8 7 3 10 3 2 7 2 10 4 6 2 6 10 8 7 86 72 75 85 ($50K) ($30K) ($35K) ($40K) Add extra criteria that boost your alternative Shave the weights of criteria that hurt your alternative Compress the scale of criteria that hurt your alternative

10. 10 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence How To Abuse Weighted Sum Methods Weight: Sports Car Sedan Mini-Van SUV 3 4 5 10 8 Fun Style Reliability Versatility Utility/K$ 2 6 10 8 7 3 10 3 2 7 2 10 4 6 /50 = /30 = /35 = /40 = 1.48 1.73 1.71 1.78 74 52 60 71

11. 11 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence The Value Of Weighted Sum Methods l Question: In a weighted sum application, can I fiddle with the weights and scores in subtle ways to make just about any alternative the winner? »Answer: Yeah, pretty much … l Question: I guess that means weighted sum methods are not very useful? »Answer: On the contrary, they are transparent and flexible … just make sure you use them properly

12. 12 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Summary: How to Use Cardinal Utility Methods l Simple weighted sum, multi-criteria utility methods are fine »MCDA methods are not a substitute for comprehensive cost- effectiveness analysis »Conduct a (pairwise) sensibility check on criteria utility tradeoffs »Don’t treat cost as another criterion, but as an overall utility divisor »Do a complete sensitivity analysis on both weights and scores to locate the decision cross-over points l Add mathematical complexity if warranted, but there is no positive value to using unnecessarily complex methods »For example, AHP calculates weights using a prescribed, vague qualitative scale in a laborious pair-wise assessment process where the DM is permitted to be inconsistent l Horses for courses: Fit the method to your problem

13. 13 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence ORDINAL METHODS l If development of a single overall utility scale is not sensible, then ordinal methods may be a viable option »If there is more than one DM (or more than one criterion being assessed) the problem is one of ‘social choice’, depending on the societal structure assumed l Nobel laureate Kenneth Arrow’s research from 1953 forms the logic foundation »“It … makes no sense to add the utility of one individual, a psychic magnitude in his mind, with the utility of another individual.” »Coke or Pepsi? … Strength of preference, or simply preference? »“A utility function whose significance lies entirely in its ordinal properties is superfluous. If we are concerned with ordinal properties it seems better to represent these directly.”

14. 14 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Kenneth Arrow and Social Welfare Functions l A ‘Social Welfare Function’ gives the ordering of alternative social states given the orderings by the individuals in the society l Arrow imposed several ‘apparently reasonable’ conditions: 1.Universality – Individuals can express any possible ordering 2.‘Welfare’ Not ‘Illfare’ - social ordering must respond positively (or at least non-negatively) to alterations in individual values 3.Independence From Irrelevant Alternatives – The societal choice amongst a set of alternatives should be unaffected if other alternatives are added or removed 4.Citizens’ Sovereignty – the social welfare function is not to be imposed (all outcomes are possible) 5.Non-Dictatorship – whenever the dictator prefers x over y, so does society l Arrow’s Theorem: You can’t have all five

15. 15 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Arrow’s Third Condition & The Voter’s Paradox l The Voter’s Paradox: »Voter 1: a b c »Voter 2: b c a »Voter 3: c a b l In this example no winner is declarable, yet if you remove any alternative from consideration a clear winner emerges l Upshot: One cannot construct an ordinal solution using the argument: “If more individuals place a ahead of b, then the consensus or social choice will place a ahead of b”.

16. 16 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence How to Abuse Ordinal Methods l Borda Count (minimum sum of rank values) is commonly employed »Simple example: three rankers, two alternatives –Ranker A: 1. Coke, 2. Pepsi –Ranker B: 1. Coke, 2. Pepsi –Ranker C: 1. Pepsi, 2. Coke »Coke wins … any and every rational ranking scheme will agree l Ranker C insists on including all contending colas (RC, Kik, …) »Ranker C tactically and artificially ranks Coke low to benefit his choice: –Ranker A: 1. Coke, 2. Pepsi, T3. Kik Cola & RC Cola –Ranker B: 1. Coke, 2. Pepsi, T3. Kik Cola & RC Cola –Ranker C: 1. Pepsi, T2. Kik Cola & RC Cola, 4. Coke l The Borda Count winner now … Pepsi!! l The Problem: Borda count fails any reasonable interpretation of the ‘independence from irrelevant alternatives’ condition l The Lesson: Don’t conduct arithmetic operations on rank values

17. 17 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence A Proposed Solution Concept to the Consensus Ranking Problem l “Given m rankings of n objects, what ranking best represents the consensus opinion?” l In its most general form: »Ties are permitted »Incomplete rankings are permitted »Relative importance of rankers is reflected (usually in form of numerical weights) l Solution Concept Employed: Select a measure of agreement between pairs of rankings and select that ranking(s) which maximizes overall average agreement. l Possible measures: »Rank correlation coefficient »Distance measure operating on the space of all rankings of n objects

18. 18 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence One Measure of Agreement: Kendall’s Tau-b (1948) 1 if object i ranked ahead of object j a ij =-1 if i ranked behind j 0 if i ranked tied with j or if i=j l Strengths: »If ties are not allowed, it is equivalent to the number of interchanges of adjacent objects required to convert one ranking into the other. l Weaknesses: »Not well defined when ties are allowed »The equivalent distance measure fails the triangle inequality test.

19. 19 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Another Agreement Measure: Kemeny-Snell Distance (1962) Axiom 1.1. d(A,B) ≥ 0, with equality holding if and only if A and B are the same ranking. Axiom 1.2. d(A,B) = d(B,A). Axiom 1.3. d(A,B) + d(B,C) ≥ d(A,C), and equality holds if and only if B is ‘between’ A and C. Axiom 2. If A' results from A by a permutation of the objects, and B' results from B by the same permutation, then d(A,B) = d(A',B'). Axiom 3. If A and B agree except for a set S of k objects, which is a segment of both, then d(A,B) may be computed as if these k objects were the only objects being ranked. Axiom 4. The minimum positive distance is 1. l K&S defined a set of axioms that any reasonable distance measure operating on the set of all orderings of n objects should satisfy … l They proved that there is a unique distance metric that satsifies these axioms, and that the following is an implemen- tation of that unique measure (using Kendall’s def’n for a ij )

20. 20 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Two Other Proposed Measures of Agreement l The Tau-X Rank Correlation Coefficient: 1 if object i ranked ahead of OR TIED WITH object j a ij =-1 if i ranked behind j 0 if i=j l The Half-Flips Metric: »Define Elemental Transformations of any ordering: 1. The Merge: Any two adjacently ranked groups of tied objects of size p and q can be merged with pq simultaneous half-flips. 2. The Split: Split any group of tied objects into two subgroups of adjacently ranked objects (of size r and s) with rs simultaneous half-flips. »The Half-Flips Metric is defined as the minimum number of half-flips required to migrate from ranking A to B

21. 21 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence A Single Unified Theory l Lemma 1. The Half-Flips Measure satisifes the Kemeny-Snell axioms. l Lemma 2. The Kemeny-Snell metric formulation is equivalent to τ X. l THEOREM. The Half-Flips Metric, the Kemeny- Snell Metric, and the τ X rank correlation coefficient are equivalent representations of the unique measure that satisfies the Kemeny-Snell axioms.

22. 22 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Max τ X – A Simple Example 0 -1 -1 1 1 0 -1 1 1 1 0 1 -1 -1 -1 0 0 1 1 1 1 0 -1 -3 1 3 0 1 1 3 1 0 0 ? ? ? ? 0 ? ? ? ? 0 ? ? ? ? 0. Max Value? Ranking Matrix l 5 Rankers (V, W, X, Y, Z ) and 4 Objects (a, b, c, d) l Rankings (all equally weighted) »V: c b a d »W: a-c d b »X: a-d b-c »Y: d b a c »Z: c-d a-b l “Combined Input Matrix”: »1*V + 1*W + 1*X + 1*Y + 1*Z = l How do we find the solution ranking(s)? »If sign matrix represents a ranking then it is the unique solution (rare!) »Could search ranking space using brute force

23. 23 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Our Simple Example d,a-c-b d,a-c,b c-d,a-b a-c-d,b c,a-d-b d-c-b,a d,c-b,a d,c,a-b c-d,b,a d,c,b,a c-d,a,b d,c,a,b c,d,a,b c,a-d,b c,b-d,a c,d,a-b a-d,c-b d,a-b,c a,c-d,b d,a,c-b a-d,b,c a-d,c,b a,d,c-b d,a,c,b a,d,b,c d-b,a-c d,b,a-c b,d,a-c b,d-c,a c-b,d,a b,c,d,a d,b,c,a d-b,c,a b,d,c,a b,d,a,c d-b,a,c d,b,a,c c,d,b,a c,b,d,a d,a,b,c a,d,c,b a-c,d,b a,c,d,b c,a,b,d c,a-b,d c,b,a,d c,b,a-d c,a,d,b c,a,b-d 7 3 2 7 3 Input Rankings (5) Solution Ranking (1) a-c-d, b Minimum Total Distance = Minimum Voter Unhappiness

24. 24 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence How Feasible Is The Brute Force Approach? l Number of rankings of n objects with ties, Q(n) »Q(2) = 3 rankings »Q(3) = 13 rankings »Q(4) = 75 rankings »Q(5) = 541 rankings »…»… l Q(n) = no. of protons on earth for what value of n? Ans. l O.A. Gross (1962) showed: 39

25. 25 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence A Cleverer Way? l Branch-and-bound algorithm devised »When implemented in FORTRAN on Pentium PC will solve for all solutions –Up to 12 or so alternatives (any number of rankers): instantly –Up to 15 or 20: seconds to minutes –Up to 20 or 25: hours to days –Over 30: basically hopeless l Heuristic algorithm developed for larger problems »Progressively applies B&B algorithm over a sliding window of 11 objects until it stabilizes »Generates solutions correctly in virtually all cases tested

26. 26 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence DISCUSSION l Extension to Incomplete Rankings »CI Matrix doesn’t change if one adds (off diagonal) zeros – this naturally represents the ‘no information’ situation between object pair »Accommodates various preferential voting schemes where objects can be ranked against other blocks of objects but are left ‘unranked’ against each other (as opposed to ‘tied’) l Difference Between Ties and Indifference »A tie is a positive statement of agreement (‘strong’ ties), not a declaration of indifference (‘weak’ ties) »Max τ X method handles both l The Advantage of Dual Theories »Extension in one theory don’t necessarily have a parallel in the other »e.g. Incomplete rankings extension in τ X domain has no parallel in Half-Flips Metric domain

27. 27 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Summary: How to Use Ordinal Methods l If development of single overall utility scale does not make sense, then you an ordinal technique may be a suitable choice l Avoid methods that involve arithmetic operations on the rank values themselves l Avoid methods that deal in strength of preference l ‘Independence from irrelevant alternatives’ is the criterion most ordinal methods run afoul of l Arrow proved that one cannot construct an ordinal solution using the argument: “If more individuals place a ahead of b, then the consensus or social choice will place a ahead of b”. l Determining consensus rankings by maximizing rank correlation using the τ X statistic is a robust method »The problem-situation must be accepting of multiple solutions »It is computationally intensive to apply

28. 28 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence

29. 29 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence How To Abuse Weighted Sum Methods The Decision: To acquire a new tank for the Army The Criteria: Weights: Alternatives: Tank A Tank B Tank C L/C Cost.2 9 3 5 Firepower.4 2 9 5 Protection.2 2 5 8 Mobility.2 7 3 5 TOTAL 1.00 4.40 5.80 5.60 B wins l Questions to ask yourself: »Are you lazy? Don’t be doing it this way when you really should be doing a comprehensive cost-effectiveness analysis. »Are you crazy? Costs are not like other utility scales; construct a sound utility scale for other factors and divide by the cost values. Change this to.3 and C wins Change it to.4 and A wins

30. 30 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Application to the Consensus Ranking Problem l Solution Concept – “Max τ X ” : Find that ranking which has maximum average correlation using τ X with all input rankings. l If each of m rankers has weight w k and input ranking matrix A k = {a k ij }, then we wish to find the solution ranking S = {s ij } that will l Or equivalently to l This permits us to create a ‘combined input matrix’ »Multiple rankers providing input on behalf of weighted criteria all roll up into a single matrix

31. 31 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Arrow, τ X, and the Consensus Ranking Problem l Kenneth Arrow’s conditions for a Social Welfare Function 1.Universality 2.‘Welfare’ Not ‘Illfare’ 3.Independence From Irrelevant Alternatives 4.Citizens’ Sovereignty 5.Non-Dictatorship l Arrow: You can’t have all five. l Emond & Mason: If you back off on #3 a bit, permitting alternatives to fall in and out of ties if other alternatives are added/removed (but not to reverse rankings in a solution), then you can have all five. l The Main Practical Problems With τ X : »It is not a well-defined Social Welfare Function in the sense that you can have multiple solutions »It is computationally intensive to apply

32. 32 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Space of All Rankings of 2 Objects a-b b,a a,b 1 half-flip

33. 33 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Space of All Rankings of 3 Objects a-b-c a-c,b b-c,a c,a-b c,b,a c,a,b a-b,c a,b-c a,c,b b,a-c b,c,a b,a,c a,b,c 1 half-flip 2 simultaneous half-flips

34. 34 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Space of All Rankings of 4 Objects d,a-c-b d,a-c,b d-c,a-b a-d-c,b c,a-d-b d-c-b,a d,c-b,a d,c,a-b d-c,b,a d,c,b,a d-c,a,b d,c,a,b c,d,a,b c,a-d,b c,d-b,a c,d,a-b a-d,c-b d,a-b,c a,d-c,b d,a,c-b a-d,b,c a-d,c,b a,d,c-b d,a,c,b a,d,b,c d-b,a-c d,b,a-c b,d,a-c b,d-c,a c-b,d,a b,c,d,a d,b,c,a d-b,c,a b,d,c,a b,d,a,c d-b,a,c d,b,a,c c,d,b,a c,b,d,a d,a,b,c a,d,c,b a-c,d,b a,c,d,b c,a,b,d c,a-b,d c,b,a,d c,b,a-d c,a,d,b c,a,d-b The geometric center of the ‘soccer ball’ represents the all-tied ranking: a-b-c-d, which is linked to the center of all eight hexagonal faces with edges of length 3, and to the center of all six square faces with edges of length 4. Length 2 edges: Length 1 edges:

35. 35 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence Sample Application: World Golf Rankings l The current weighted points system is very complicated l Max τ X is applicable to this problem: »Every significant tournament over last 2 years (180) is a ‘voter’ for the rank of participating players –Tournament rankings are necessarily ‘incomplete’ rankings –Ties between players making the cut are ‘strong’ –Ties between players who missed cut, were disqualified, or withdrew are ‘weak’ »Can use same tournament weight values as current system, based on time elapsed (starts at 8, decrementing by 1 every 13 weeks) »No extra weighting of the 4 ‘Majors’ required, as tournaments with heavy Top 50 participation naturally weight themselves l Minimize total weighted ‘unhappiness’ of the 180 voters …

36. 36 Operational Research Division / Division de la Recherche Opérationnelle Défense nationale National Defence World Golf Ranking and Max τ X Comparison 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. Woods Els Singh Mickelson Love Goosen Weir Harrington Furyk Garcia Perry Scott Campbell Clarke Appleby Hamilton Verplank Ames Toms Haas Woods Els Singh Mickelson Love Goosen Weir Harrington Furyk Garcia Perry Scott Campbell Clarke Appleby Hamilton Verplank Ames Toms Haas Aug 12 World Ranking 2. 1. 3. 4. 5. 6. T9. 7. 8. 14. T9. T17. 16. 15. 19. T11. 20. T11. 2. 1. 3. 4. 5. 6. T9. 7. 8. 14. T9. T17. 16. 15. 19. T11. 20. T11. Max τ X Ranking