ON ELICITATION TECHNIQUES OF NEAR-CONSISTENT PAIRWISE COMPARISON MATRICES József Temesi Department of Operations Research Corvinus University of Budapest, Hungary IFORS 2011, Melbourne, Australia

2 OUTLINE  Pairwise Comparison Matrices (PCM)  Some properties of PCMs  Empirical research on PCMs  Elicitation and adjustment methods  Interactive procedures: recommendations  References

IFORS 2011, Melbourne, Australia 3 The decision-maker (DM) estimates unknown weights of criteria or preference values of certain alternatives with respect to a given criterion using the method of pairwise comparisons. w 1, w 2, …, w n : implicit weights, a ij ( a ii = 1, a ij > 0, and a ij = 1/a ji, ; i, j = 1,…,n): DM’s evaluation for every pair. A is a positive, consistent, reciprocal matrix if a ij a jk = a ik for each i, j, k = 1, …, n, and the rank of A is 1. PAIRWISE COMPARISON MATRICES (PCM)

IFORS 2011, Melbourne, Australia 4

5

6 Error-free property and consistency The PCM is error-free if a ij = w i /w j (for all i, j = 1, 2, …, n, i  j ), as defined by Choo and Wedley (2004). In that case the elements of the PCM reflect the real preferences of the DM. If the PCM is error-free, then it is consistent. If the elements of the PCM are given on ratio scale, the decision maker is error-free if (i)the PCM is a positive, reciprocal, consistent matrix, (ii)the pairwise comparisons reflect precisely to the decision maker’s real preferences. The decision maker is error-free if and only if the PCM is error-free (Temesi, 2011).

IFORS 2011, Melbourne, Australia 7 Inconsistency The usual assumption is that the DM is not consequent in her revealed comparisons and in real cases the values of a ij and w i /w j differ from each other. In a real case we can say that the consistent behavior of a DM means that the DM’s pairwise comparison matrix is as close to the one rank matrix as possible (near-consistency). There is a difference between a naïve decision maker and an expert. There is a need for inconsistency ratios, indices of a PCM.

IFORS 2011, Melbourne, Australia 8 Saaty’s inconsistency ratio ( CR ) (Saaty, 1980)

IFORS 2011, Melbourne, Australia 9 Koczkodaj’s approach (Koczkodaj, 1993)

IFORS 2011, Melbourne, Australia 10 Duszak and Koczkodaj (1994) extended this definition for a general n × n reciprocal matrix A as the maximum of CM(a, b, c) for all triads (a, b, c), i.e., 3 × 3 submatrices which are themselves pairwise comparison matrices, in A: CM(A) = max{CM(a ij, a ik, a jk ) | 1  i < j < k  n} An extension of CM inconsistency for higher dimensions

IFORS 2011, Melbourne, Australia 11 AN EXPERIMENT A pool of empirical pairwise comparison matrices has been generated. Properties have been analyzed (Bozóki-Dezső-Poesz-Temesi, 2011):  inconsistency  rank reversal  sensitivity of the weight vector Dimensions of the experiments:  Type of the problem  Size of the problem  Questioning method

12 Type of the problem: subjective (summer houses) A B C D E F IFORS 2011, Melbourne, Australia

13 Type of the problem: objective (map)

IFORS 2011, Melbourne, Australia 14 Questioning methods sequential random Ross

IFORS 2011, Melbourne, Australia 15 Size of the problem: 4 × 4, 6 × 6, 8 × 8 Special dimension of the research: complete and incomplete PCMs

IFORS 2011, Melbourne, Australia 16 EMPIRICAL RESEARCH: testing environment Controlled tests Subjects: Bachelor and Master students Group sizes: persons, total number is 227 Anonymity: codes 2 exercises in each group; number of matrices to be analyzed: 454

IFORS 2011, Melbourne, Australia 17 Research questions Q1. Are inconsistency indices systematically higher in case of subjective type of problems? A: Yes Q2. Are inconsistency indices higher in case of large size PC matrices? A: Yes Q3. Has the questioning method an impact on the inconsistency? A: No Q4. Is the behavior of the decision maker consequent in the course of the whole questioning procedure? A: Yes, for most of the decision makers Q5. What can we say about inconsistency and the weight vector if both are computed from incomplete data? A: it is possible to use incomplete matrices to estimate the final scores and weights

IFORS 2011, Melbourne, Australia 18 Individual inconsistency levels through the filling in process

IFORS 2011, Melbourne, Australia 19 number of matrix elements Order Sequential0,000,961,823,714,745,666,617,338,359,2110,75 Random0,011,382,773,494,424,976,256,918,178,199,47 Ross0,011,372,503,844,935,456,277,247,859,5210,63 number of matrix elements Order Sequential0,000,130,180,250,320,400,480,550,640,720,81 Random0,010,060,110,210,400,510,580,660,720,730,86 Ross0,000,070,140,230,310,370,500,730,790,890,94 The average of CR inconsistencies (in %) in case of 6×6 incomplete matrices Summer houses Maps

IFORS 2011, Melbourne, Australia 20 number of matrix elements Type Summer houses0,820,880,900,920,930,940,960,97 0,981,00 Maps0,97 0,98 0,99 1,00 Spearman rank correlation coefficients in case of 6×6 incomplete matrices

IFORS 2011, Melbourne, Australia 21 Biases in the comparisons Type 1: The errors, fluctuations in the judgments of the DM are around the real values within a small distance; there is a kind of “noise” in the system. Ex-post correction with a perturbation method. Type 2: Systematic errors, under- or overestimations of certain comparisons (e.g. for a subset of alternatives) occur. Correction of a sequence of triads ex-post or during the procedure. Type 3: Outliers could be detected, e.g. as a consequence of communication errors or of the mistake of the DM. Immediate or ex-post correction.

IFORS 2011, Melbourne, Australia 22 Conclusions to create new questioning procedures Decision-aiding is possible and meaningful. Aim: to produce near-consistent PCM. Assumptions:  rationale decision-maker  interactions are viable during the questioning method (e.g. on-line questioning model)  built-in methods to handle incomplete matrices

IFORS 2011, Melbourne, Australia 23 Approaches Warning system: based on the experimental conclusions in case of errors Type 2 and Type 3. Advantage: near-consistent matrix can be reached. Stopping rule: to use an incomplete matrix with enough comparisons to calculate all elements of a near-consistent complete PCM. Advantage: large and/or subjective type problems can be handled with a reduced number of comparisons. Our reserach team will conduct further experiments.

IFORS 2011, Melbourne, Australia 24 REFERENCES Bozóki, S., Fülöp, J., Rónyai,L. [2010]: On optimal completions of incomplete pairwise comparison matrices, Mathematical and Computer Modelling, 52, pp Bozóki, S., Fülöp, Koczkodaj, W.W. [2011]: An LP-based inconsistency monitoring of pairwise comparison matrices, Mathematical and Computer Modelling, 54(1-2), pp Bozóki, S., Dezső, L., Poesz, A., Temesi, J. [2011]: Pairwise comparison matrices: an empirical research, Proceedings of the International Symposium on the AHP for MCDM, Sorrento, Naples, Italy, June15-18, 2011, Online Proceedings ISSN

IFORS 2011, Melbourne, Australia 25 Choo, E.U., Wedley, W.C. [2004] A common framework for deriving preference values from pairwise comparison matrices, Computers & Operations Research 31, 893–908. Duszak, Z., Koczkodaj, W.W. [1994]: Generalization of a new definition of consistency for pairwise comparisons, Information Processing Letters, 52, pp. 273–276. Koczkodaj, W.W. [1993] A new definition of inconsistency of pairwise comparisons, Mathematical and Computer Modelling 8, pp Saaty, T.L. [1980]: The analytic hierarchy process, McGraw Hill, N.Y. Temesi, J.: [2011]:Pairwise comparison matrices and the error-free property of the decision-maker, Central European Journal of Operations Research, Vol. 19. No 2. June 2011, pp.