1. S ystems Analysis Laboratory Helsinki University of Technology Can We Avoid Biases in Environmental Decision Analysis ? Raimo P. Hämäläinen Helsinki University of Technology Systems Analysis Laboratory raimo@hut.fi www.paijanne.hut.fi

2. S ystems Analysis Laboratory Helsinki University of Technology Structure of the presentation Background & decision analysis interviews Goals of the study Case: Regulation of Lake Päijänne Splitting bias & swapping of levels Description of the experiment Results of the experiment Conclusions ?

3. S ystems Analysis Laboratory Helsinki University of Technology Environmental decision analysis Parliamentary nuclear power decision (Hämäläinen et. al) Decision analysis interviews (Marttunen & Hämäläinen) Spontaneous decision conferencing in nuclear emergency management (Hämäläinen & Sinkko)

4. S ystems Analysis Laboratory Helsinki University of Technology Cognitive biases Splitting bias –attribute receives more weight if it is split –origins: subjects give rank information only (Pöyhönen & Hämäläinen) –Not observable in hierarchical weighting

5. S ystems Analysis Laboratory Helsinki University of Technology Decision analysis interviews Opinions of large groups of people traditionally collected through questionnaires Decision analysis interviews may provide a more reliable way to collect these opinions Idea: –one value tree for all = common terminology –emphasis on finding the viewpoints of different stakeholder groups –interactive, computer supported

6. S ystems Analysis Laboratory Helsinki University of Technology Research interest Existence of biases in a real case Can biases can be avoided through training and proper instructing ? Identify what can go wrong in the Lake Päijänne case Compare the well trained university students’ and spontaneous stakeholders’ responses

7. S ystems Analysis Laboratory Helsinki University of Technology The Lake Päijänne case Regulation started 1964 Main aims were to improve hydroelectricity production and to reduce damages caused by flooding Environmental values & increase in free time need for an improved regulation policy

8. S ystems Analysis Laboratory Helsinki University of Technology Splitting bias When an attribute is split, the weight it receives increases 0.4 0.3 0.1 0.3 0.4

9. S ystems Analysis Laboratory Helsinki University of Technology Swapping of levels Does the order of the levels affect the resulting weights? Important question in environmental decision analysis: –stakeholder groups may vary regionally Not studied before

10. S ystems Analysis Laboratory Helsinki University of Technology Example of swapping of levels Lake Päijänne River Kymijoki Attribute 3 Attribute 2 Attribute 1 Attribute 2 Attribute 1 Attribute 3 Attribute 2 Attribute 1 River Kymijoki Lake Päijänne River Kymijoki Lake Päijänne River Kymijoki Lake Päijänne

11. S ystems Analysis Laboratory Helsinki University of Technology Earlier experiments on biases Structure of the decision model affects the results Previous experiments typically: –subjects: university students –problems: artificial –results: taken from group averages Lake Päijänne-case: a real problem with real stakeholders

12. S ystems Analysis Laboratory Helsinki University of Technology Important new features Realistic case Decision analysis interviews instead of passive decision support or survey Interactive computer support (resulting weights shown immediately) Instructions and training before the weighting

13. S ystems Analysis Laboratory Helsinki University of Technology Subjects: University students attending a course on decision analysis (N = 30) –held during a tutorial session, not mandatory Habitants of Asikkala (N = 40) –3 groups of students –1 group of adults (volunteers) 3 experts from the Finnish Environment Institute & 2 summer residence owners

14. S ystems Analysis Laboratory Helsinki University of Technology Experimental setting Weighting done with the SWING method using a tailored Excel interface Subjects entered the numbers themselves, two assistants were present to help Resulting weights shown as bars Order of value trees partly randomized

15. S ystems Analysis Laboratory Helsinki University of Technology Sessions A short introduction to: –Lake Päijänne case –value trees & weighting –different structures of the value tree In HUT the avoidance of biases was emphasized more Duration: 60 - 90 minutes

16. S ystems Analysis Laboratory Helsinki University of Technology SWING method Easy to use Attribute ranges clearly presented Idea: –choose the attribute you would first like to move to its best level –assign it 100 points –assign other attributes points less than 100 in respect to the first attribute

17. S ystems Analysis Laboratory Helsinki University of Technology Flat-weighting Muu talous ??? Vesivoima Muu talous Ympäristö Talous Rantojen käytettävyys Virkistyskalastus Kalojen lisääntyminen Rantakasvillisuus Lahtien umpeenkasvu Virkistys Luonto Tulvat, maatalous ja teollisuus Tulvat, loma-asutus Vesiliikenne Ammattikalastus

18. S ystems Analysis Laboratory Helsinki University of Technology Upper level weights: Muu talous ??? Vesivoima Muu talous Ympäristö Talous Rantojen käytettävyys Virkistyskalastus Kalojen lisääntyminen Rantakasvillisuus Lahtien umpeenkasvu Virkistys Luonto Tulvat, maatalous ja teollisuus Tulvat, loma-asutus Vesiliikenne Ammattikalastus

19. S ystems Analysis Laboratory Helsinki University of Technology ENV5-tree: Luonto Virkistys Ympäristö Talous Rantojen käytettävyys Virkistyskalastus Kalojen lisääntyminen Rantakasvillisuus Lahtien umpeenkasvu

20. S ystems Analysis Laboratory Helsinki University of Technology ENV2-tree: Luonto Virkistys Ympäristö Talous

21. S ystems Analysis Laboratory Helsinki University of Technology EC5-tree: Muu talous ??? Vesivoima Muu talous Ympäristö Talous Tulvat, maatalous ja teollisuus Tulvat, loma-asutus Vesiliikenne Ammattikalastus

22. S ystems Analysis Laboratory Helsinki University of Technology EC2-tree: Muu talous ??? Vesivoima Muu talous Talous Muu talous ??? Vesivoima Muu talous Ympäristö Talous

23. S ystems Analysis Laboratory Helsinki University of Technology Swapping of levels: Muu talous ??? Kymijoki ja muut Päijänne Rantakasvillisuus Tulvavahingot Kymijoki ja muut Päijänne Muu talous ??? Rantakasvillisuus Tulvavahingot Kymijoki ja muut Päijänne Rantakasvillisuus Tulvavahingot

24. S ystems Analysis Laboratory Helsinki University of Technology Flat weights vs. upper level weights Both in group averages and in results of individuals the total weights for the environment and economy were similar with both methods One explanation: symmetric value tree

25. S ystems Analysis Laboratory Helsinki University of Technology Splitting bias

26. S ystems Analysis Laboratory Helsinki University of Technology A typical resident in Asikkala ENVIRONMENTECONOMY 5 1 5 2 1 1 5 1 1 1 5 2

27. S ystems Analysis Laboratory Helsinki University of Technology Example from HUT (one of the best ones) ENVIRONMENTECONOMY 5 1 5 2 1 15 1 1 1 5 2

28. S ystems Analysis Laboratory Helsinki University of Technology Why even weights ? Some students: none of the attributes seemed to be important Asikkala: all of the attributes were important even weights for all attributes

29. S ystems Analysis Laboratory Helsinki University of Technology What caused the bias ? Similar points for all attributes in one branch regardless of the structure of the value tree

30. S ystems Analysis Laboratory Helsinki University of Technology Effect of instructions Students had good instructions – only some had bias in their results In the spontaneous stakeholders’ sessions the information load was too high and thus the instructions were not adopted as well – nearly all had systematically consistent bias

31. S ystems Analysis Laboratory Helsinki University of Technology STUDENTSSTAKEHOLDERS Adjusted / not adjusted weights

32. S ystems Analysis Laboratory Helsinki University of Technology Examples STUDENTSSTAKEHOLDERS

33. S ystems Analysis Laboratory Helsinki University of Technology Observation The students and the experts from FEI could nearly avoid the splitting bias –good background education + instructions did reduce the bias What did the students think? - Arithmetics or real avoidance of biases

34. S ystems Analysis Laboratory Helsinki University of Technology Avoiding the splitting bias ? Good instruction can eliminate it When the economical attributes were split, the magnitude of the bias was slightly larger Graphical feedback did not eliminate Hierarchical weighting

35. S ystems Analysis Laboratory Helsinki University of Technology Swapping of attribute levels If the order of the levels would not affect the weigts, the pairs of weights should be equal (as in the first picture)

36. S ystems Analysis Laboratory Helsinki University of Technology Conclusions about swapping of levels ? Only a few had clearly differing weights with the two trees No systematic pattern was found Less differences residents of Asikkala and students than with the splitting bias A simple scale lead to similar weights with both trees (100, 70 for example) Neither tree gained clear support

37. S ystems Analysis Laboratory Helsinki University of Technology Solutions to reduce biases ? Hierarchical weighting Models should be tested on real decision makers Interactiveness of weighting (= possibility to return to change the points given earlier ) Well balanced trees

38. S ystems Analysis Laboratory Helsinki University of Technology Other observations in Asikkala Concept of weight seemed to be difficult for most subjects in Asikkala Information load was high Facilitators role becomes important when the DM’s are uncertain

39. S ystems Analysis Laboratory Helsinki University of Technology Problems related to the Lake Päijänne case Current regulation policy cannot be improved very significantly –no big differences between the alternatives –unrealistic hopes and false information are probably larger problems than the regulation itself ‘money is not money’ –strong feelings against the power companies and regulation (shape of value function ?)

40. S ystems Analysis Laboratory Helsinki University of Technology Suggestions for future research Hierarchical weighting Encouragement to reconsider and readjust the statements iterate Decision Analyst must supervise!

41. S ystems Analysis Laboratory Helsinki University of Technology R.P. Hämäläinen, E. Kettunen, M. Marttunen and H. Ehtamo: Evaluating a framework for multi-stakeholder decision support in water resources management, Group Decision and Negotiation, 2001. (to appear) M. Pöyhönen, Hans C.J. Vrolijk and R.P. Hämäläinen: Behavioral and procedural consequences of structural variation in value trees. European Journal of Operational Research, 2001. (to appear) M. Pöyhönen and R.P. Hämäläinen: There is hope in attribute weighting, Journal of Information Systems and Operational Research (INFOR), vol. 38, no. 3, Aug. 2000, pp. 272-282. Abstract R.P. Hämäläinen, M. Lindstedt and K. Sinkko: Multi-attribute risk analysis in nuclear emergency management, Risk Analysis, Vol. 20, No 4, 2000, pp. 455-467. M. Pöyhönen and R.P. Hämäläinen: Notes on the weighting biases in value trees, Journal of Behavioral Decision Making, Vol. 11, 1998, pp. 139-150. Susanna Alaja: Structuring effects in environmental decision models, Helsinki University of Technology, Systems Analysis Laboratory, Theses, 1998. References

42. S ystems Analysis Laboratory Helsinki University of Technology M. Pöyhönen, R.P. Hämäläinen and A. A. Salo: An experiment on the numerical modeling of verbal ratio statements, Journal of Multi-Criteria Decision Analysis, Vol. 6, 1997, pp. 1-10. R.P. Hämäläinen and M. Pöyhönen: On-line group decision support by preference programming in traffic planning, Group Decision and Negotiation, Vol. 5, 1996, pp.485-50. M. Marttunen and R.P. Hämäläinen: Decision analysis interviews in environmental impact assessment, European Journal of Operational Research, Vol. 87, No. 3, 1995, pp. 551-563. R.P. Hämäläinen, A.A. Salo and K. Pöysti: Observations about consensus seeking in a multiple criteria environment, in: Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences, Vol. IV, 1991, IEEE Computer Society Press, Hawaii, pp. 190-198. R.P. Hämäläinen: Computer assisted energy policy analysis in the parliament of Finland, Interfaces, Vol. 18, No. 4, 1988, pp. 12-23. Also in: Case and Readings in Management Science, 2nd edition, M. Render, R.M. Stair Jr. and I. Greenberg (eds.), Allyn & Bacon, Massachusetts 1990 pp. 278-288.