1. S ystems Analysis Laboratory Helsinki University of Technology Observations from computer- supported Even Swaps experiments using the Smart-Swaps software Jyri Mustajoki Raimo P. Hämäläinen Petri Lievonen Systems Analysis Laboratory Helsinki University of Technology www.sal.hut.fi

2. S ystems Analysis Laboratory Helsinki University of Technology Introduction Even Swaps method Hammond, Keeney and Raiffa (1998, 1999) Easy-to-use multi-criteria decision analytical (MCDA) method based on value tradeoffs Web-based Smart-Swaps software Procedural support to carry out even swaps In this study, observations of students using the method with the help of the software Do they like and understand it? How laborious it is felt to be? Role and importance of the computer support?

3. S ystems Analysis Laboratory Helsinki University of Technology Structure of this presentation Introduction to the Even Swaps process and the Smart-Swaps software Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure Results and evidence Conclusions and discussion

4. S ystems Analysis Laboratory Helsinki University of Technology PrOACT-model Smart Choices (1999) Define your decision problem to solve the right problem. Clarify what you’re really trying to achieve with your decision. Make smarter choices by creating better alternatives to choose from. Describe how well each alternative meets your objectives. Make tough compromises when you can’t achieve all your objectives at once. Pr oblem O bjectives A lternatives C onsequences T radeoffs Smart- Swaps software - this experiment Introduction to the Even Swaps process and the Smart-Swaps software + Uncertainty, Risk profiles, Linked decisions

5. S ystems Analysis Laboratory Helsinki University of Technology Even Swaps elimination process Carry out even swaps that make a)Alternatives dominated (attribute-wise) ≡There is another alternative, which is equal or better than this in every attribute, and better at least in one attribute b)Attributes irrelevant ≡Each alternative has the same value on this attribute »These can be eliminated Process continues until one alternative, i.e. the best one, remains Introduction to the Even Swaps process and the Smart-Swaps software

6. S ystems Analysis Laboratory Helsinki University of Technology Smart-Swaps software www.smart-swaps.hut.fi Introduction to the Even Swaps process and the Smart-Swaps software

7. S ystems Analysis Laboratory Helsinki University of Technology Example Office selection problem (Hammond et al. 1999) Dominated by Lombard Practically dominated by Montana (Slightly better in Monthly Cost, but equal or worse in all other attributes) 78 25 An even swap Commute time removed as irrelevant Introduction to the Even Swaps process and the Smart-Swaps software

8. S ystems Analysis Laboratory Helsinki University of Technology Supporting Even Swaps with Preference Programming Support for 1.Finding candidates for the next even swap 2.Identifying practically (i.e. almost) dominated alternatives Both tasks need comprehensive technical screening Idea: supporting the process – not automating it Introduction to the Even Swaps process and the Smart-Swaps software

9. S ystems Analysis Laboratory Helsinki University of Technology Decision support in Smart-Swaps More than one remaining alternative The most preferred alternative is found Trade-off information Even swap suggestions Practical dominance candidates Initial statements about the attributes Problem initialization Updating of the model Make an even swap Eliminate irrelevant attributes Eliminate dominated alternatives Yes No Even Swaps Preference Programming Introduction to the Even Swaps process and the Smart-Swaps software

10. S ystems Analysis Laboratory Helsinki University of Technology Previous studies of Even Swaps Comparison of Even Swaps and MAVT Belton et al. (2005): “MCDA in E-democracy. Why weight? Comparing Even Swaps and MAVT.”, TED Workshop, May 19-22, Helsinki Environmental planning Gregory et al. (2001): “Bringing stakeholder values into environmental policy choices: a community-based estuary case study.” Ecological Economics 39, 37-52. Strategy selection in a rural enterprise Kajanus et al. (2001): “Application of even swaps for strategy selection in a rural enterprise.” Management Decision 39(5), 394-402. Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

11. S ystems Analysis Laboratory Helsinki University of Technology Thoughts about Even Swaps process From a cognitive point of view, Even Swaps process has several characteristics that are of a special interest, e.g. What kind of swaps the DMs tend to carry out? How the DMs understand the alternatives with the revised consequences? Does the DM end up with the same result following different paths of even swaps? Smart-Swaps software available »Easy to study how the DMs carry out the process in practice Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

12. S ystems Analysis Laboratory Helsinki University of Technology Questions of interest in our experiment In this study, the focus is on: 1)Issues related to carrying out the process with the Smart-Swaps software or (manually) with Microsoft Excel? 2)Issues related to the size of the problem? 3)What are the benefits of using the Preference Programming functionality of Smart-Swaps software, if any? Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

13. S ystems Analysis Laboratory Helsinki University of Technology Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure Material and methods Subjects consisting of engineering students as DMs Controlled experiments included in courses 1)Assignments in applied mathematics 2)Decision making and problem solving Questionnaires with open and scaled questions Decision process logs

14. S ystems Analysis Laboratory Helsinki University of Technology Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure Experimental procedure After a brief introduction to the Even Swaps process each DM carried out two decision analytical assignments: 1)The Even Swaps process on a small introductory problem. On this assignment, A.half of the DMs used Excel manually B.the other half conducted the process with the Smart- Swaps software 2)The Even Swaps process on a much larger problem. Every DM used the Smart-Swaps software but A.half of the DMs were instructed to ignore the Preference Programming functionality (‘recommender’) B.the other half was instructed to utilize it

15. S ystems Analysis Laboratory Helsinki University of Technology Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure 1 st : a small introductory problem You are choosing an apartment. After preliminary elimination there are three alternatives: Lombard, Baranov and Montana. You are in a happy position, as all want you as a tenant. It all boils down to what do you want. You have ended up with four criteria to base your decision on. You gather up a consequences table: LombardBaranovMontana Distance to school25 min20 min45 min ConditionPoorGoodExcellent Size35 m 2 42 m 2 23 m 2 Rent350 €/month450 €/month250 €/month

16. S ystems Analysis Laboratory Helsinki University of Technology Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure 2 nd : a large primary problem Your company is about to choose the facility for a new office. After serious thinking you consider eight attributes relevant: a1: size of the office a2: rental costs a3: renovation need a4: car park opportunities a5: means of commuting a6: distance to city center a7: other facilities in the neighborhood a8: habitability After searching for a while you find 12 alternatives to choose from (X1-X12). The consequences are given in a table.

17. S ystems Analysis Laboratory Helsinki University of Technology 1 st study 2 nd study both 1. Small problem A (Excel) 10 DMs6 DMs16 pers. B (Smart-Swaps) 10 DMs14 DMs24 pers. 2. Large problem A (without recomm.) 10 DMs9 DMs19 pers. B (with recommender) 10 DMs11 DMs21 pers. Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure Overall sample sizes Small but sufficient for our purposes to get preliminary results.

18. S ystems Analysis Laboratory Helsinki University of Technology Hypotheses on the experiment H1.1: The DMs using the Smart-Swaps software evaluate the results more positively than the DMs using Excel. H1.2: The DMs using the Smart-Swaps require less time to get the result than the DMs using Excel. H2.1: The support by Smart-Swaps is evaluated to be more useful in large problems than in small ones. H2.2: The DMs tend to eliminate rather dominated alternatives than irrelevant attributes. H3.1: The even swap recommender is evaluated as useful. H3.2: Utilizing the recommender results in fewer swaps. H3.3: Practical dominance propositions are seldom neglected. Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

19. S ystems Analysis Laboratory Helsinki University of Technology DMs arrived to various decisions Results and evidence Decisions in the 1 st (small) and 2 nd (large) problems. Areas indicate frequencies of subjects that chose certain alternative.

20. S ystems Analysis Laboratory Helsinki University of Technology H1: Smart-Swaps software vs. Excel? Results and evidence The DMs using Smart-Swaps were faster No significant difference between no. of swaps Note that the problem was small No significant difference between decisions made No significant difference between the opinions on the result

21. S ystems Analysis Laboratory Helsinki University of Technology The DMs using Smart-Swaps were faster Results and evidence Decision time comparison in the 1 st (small) problem. Areas indicate frequencies of subjects that completed the task in certain time. Smart-Swaps software vs. Excel:

22. S ystems Analysis Laboratory Helsinki University of Technology No significant difference between no. of swaps Number of swaps on the 1 st problem. Areas indicate frequencies of subjects that completed the task with certain amount of swaps. Results and evidence Smart-Swaps software vs. Excel:

23. S ystems Analysis Laboratory Helsinki University of Technology H2: Issues related to the size of the problem In large problems, it may be very difficult to manually carry out the even swaps process Smart-Swaps is regarded as helpful in applying the Even Swaps process also in large problems DMs tend to eliminate more dominated alternatives than irrelevant attributes Results and evidence

24. S ystems Analysis Laboratory Helsinki University of Technology The Even Swaps process is seen to apply best to small problems Results and evidence Areas indicate frequencies of subjects that gave certain answer on Osgood-scale. Size of the problem:

25. S ystems Analysis Laboratory Helsinki University of Technology Elimination by dominance was used more than irrelevance Results and evidence Number of all eliminations in the 2 nd problem studies are presented in the table. Size of the problem: No proposerWith proposer ALLALL-%AA-%B B-% Removal6611%5321%134% Dominance34559%13052%21564% Irrelevance16328%5823%10531% Practical dom.112%73%41% 585100%248100%337 100%

26. S ystems Analysis Laboratory Helsinki University of Technology H3: Benefits of using Smart-Swaps The DMs using the Preference Programming functionality (the recommender) were faster The DMs using the recommender made fewer swaps The recommender was rated high The DMs using the recommender gave more positive opinions on the method Results and evidence

27. S ystems Analysis Laboratory Helsinki University of Technology DMs using the proposer were faster Results and evidence Decision time comparison in the 2 nd (large) problem. Areas indicate frequencies of subjects that completed the task in certain time. Benefits of Smart-Swaps:

28. S ystems Analysis Laboratory Helsinki University of Technology DMs using the proposer made fewer swaps Results and evidence Number of swaps on the 2 nd problem. Areas indicate frequencies of subjects that completed the task with certain amount of swaps. Benefits of Smart-Swaps:

29. S ystems Analysis Laboratory Helsinki University of Technology Preference programming functionality was rated high Results and evidence Benefits of Smart-Swaps: Opinions on the recommender functionality of the Smart-Swaps software. Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.

30. S ystems Analysis Laboratory Helsinki University of Technology DMs without proposer vs. DMs using proposer Results and evidence Benefits of Smart-Swaps: Differences in opinions between DMs ignoring the recommender and DMs utilizing it. Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.

31. S ystems Analysis Laboratory Helsinki University of Technology DMs without proposer vs. DMs using proposer Results and evidence Benefits of Smart-Swaps:

32. S ystems Analysis Laboratory Helsinki University of Technology Conclusions Our first experiments on the Smart-Swaps software suggest: The support provided by the Smart-Swaps software is perceived to be useful Preference Programming approach provided considerable help A significant difference between the opinions of the group utilizing the recommender and the group instructed to ignore it »Further studies needed to get a deeper insight of how the DMs perceive the approach in practice »In addition, we got a lot of usability feedback to help improve the Smart-Swaps software Conclusions and discussion

33. S ystems Analysis Laboratory Helsinki University of Technology References V. Belton, G. Wright and G. Montibeller (2005). “MCDA in E-democracy. Why weight? Comparing Even Swaps and MAVT.” Presentation at the TED Workshop on e- Participation in Environmental Decision Making, May 2005, Helsinki. (Downloadable at http://www.ted.tkk.fi/presentations/Belton-TED.ppt) Hammond, J.S., Keeney, R.L., Raiffa, H., 1998. “Even swaps: A rational method for making trade-offs.” Harvard Business Review 76(2), 137-149. Hammond, J.S., Keeney, R.L., Raiffa, H., 1999. Smart choices. A practical guide to making better decisions. Harvard Business School Press, Boston. Mustajoki, J., Hämäläinen, R.P., 2005. “A Preference Programming Approach to Make the Even Swaps Method Even Easier.” Decision Analysis. (to appear) (Downloadable at www.sal.hut.fi/Publications/pdf-files/mmus04.pdf) Salo, A., Hämäläinen, R.P., 1992. “Preference assessment by imprecise ratio statements.” Operations Research 40(6), 1053-1061. A. Salo and R.P. Hämäläinen (1995). ”Preference programming through approximate ratio comparisons.” European Journal of Operational Research 82(3), 458-475. Applications of Even Swaps: Gregory, R., Wellman, K., 2001. “Bringing stakeholder values into environmental policy choices: a community-based estuary case study.” Ecological Economics 39, 37-52. Kajanus, M., Ahola, J., Kurttila, M., Pesonen, M., 2001. “Application of even swaps for strategy selection in a rural enterprise.” Management Decision 39(5), 394-402.