1. 1 Delphi for Foresight and Large Group Communications Murray Turoff turoff@njit.edu turoff@njit.edu http://is.njit.edu/turoff

2. © 2009 Turoff 2 Definition from 1975 Design of a group communication process structured/tailored around the nature of the application and the nature of the group Original paper and pencil rounds Anyone can change their view Anonymity or pennames Scaling theory to promote understanding Voting to focus discussion Select “knowledgeable” people A round took a month – three to five rounds Respondents 15 to 500 Prediction, policy analysis, conditional forecasts, planning, significance of contributions, new product characteristics, etc, etc. (book has many examples)

3. © 2009 Turoff 3 Added definitions and analogs today A rose by any other name! Recommender systems Marketplace Systems Collaborative Systems Knowledge structures and systems Collaborative tagging and Folksonomies ESPgame.org A beautifully simple Delphi process Structural Modeling (user created models) Computer Mediated Communications The design of inquiry systems Scientific: Leibnitz, Locke, Kant, Hegel, Singer Non Scientific! Heidigger (Negotiated reality and marketing) Virtual Organizations, teams, groups

4. © 2009 Turoff 4 Delphi does not make decisions Delphi should never make a decision, it should be used to analyze a potential decision or action and provide that to a single individual or role that is accountable for the decision to be made. The decision maker for real time decision should always be one of the participants In Emergency management many different roles make decisions that can interfere with one another and an Emergency System must track the decision process to insure everyone has access to the most current relevant information This type of system alerts people to conflicts and those responsible for oversight. The problem solving Delphi (later) is useful in that type of situation. It is unpredictable what Roles (active 24 hours 7 days a week) are needed at a given time for a given problem but it is always a group that is needed.

5. © 2009 Turoff 5 Characteristics of Delphi I A content structure appropriate to the problem being addressed Automatic organization of content Dynamic knowledge base creation Anonymity of voting and degrees of anonymity for contributions Forced, voluntary, and pen names Some groups are told who other members are to convince them of quality of group (peers) Anyone can change their original vote Primary goal of votes to expose different views and generate discussion of causes Ambiguities, uncertainties, misunderstandings, and disagreements

6. © 2009 Turoff 6 Characteristics of Delphi II Classifying participants so that votes can be seen viewed by backgrounds of voters very important for large studies with very heterogeneous groups Use of voting and comments is to encourage participants to change their views based upon relevant contributions of others. This is a different objective than surveys and associated assumption about only measuring a fixed state of mind (psychological characteristics) that do not change for an adult Not all survey guidelines and associated statistics apply to a process intended to deal with complex problems and causing people to rethink their viewpoint Pennames extremely useful for allowing continuity of thought across a set of comments and for execution of role playing learning games or story telling For example: comment on the following policy resolutions as the company auditor would

7. © 2009 Turoff 7 Ultimate Delphi Goal: Collective Intelligence Group problem solving result is better than any single member could have done before the discussion process Possible with even paper and pencil Group can solve problems as fast as any single member considering the same relevant materials (asynchronous operation) Possible with computers and Structured Computer Mediated Communications

8. © 2009 Turoff 8 Creativity Experiment Open ended idea generation Distribution of total Raw, Unique, and Rare Ideas Online Discussion System alone and with Delphi Voting imposed x group size. Problem: product ideas for a pill sized data responder, ranked list report. Notes: Unique ideas (duplications removed), rare ideas (occurred in 3 or less groups); Statistically significant on per group and per person that Delphi leads to more improvisation or creativity. Cho, H. (2004) The effect of Delphi structure on small and medium-sized asynchronous groups, Ph. D. Dissertation, January 2004, New Jersey Institute of Technology, Information Systems Department. http://www.library.njit.edu/etd/index.cfm

9. © 2009 Turoff 9 Number of Idea Types Structure x Group Size 44 groups (11 groups in each condition) Small 5-7 persons Medium 11-14 persons Delphi: Raw ideas (duplicates)157247 Unique ideas67111 Rare ideas (g<=3))4057 Unstructured: Raw ideas (duplicates)108192 Unique ideas5286 Rare ideas (g<=3)1948 Results for total ideas and ideas per person are both statistically significant

10. © 2009 Turoff 10 Easy Online Delphi Bulletin Board System imposed structure by monitor Good discussion thread structure Root items for main ideas Replies for discussions of each idea Allowed anonymous or real signatures Web survey system where monitor put the ideas for a vote and provided summaries online Once a week for three weeks Non structured condition Same system but no imposed structure by monitor and no surveys done

11. © 2009 Turoff 11 Why Delphi Too many participants needed for face-to-face Three to five in any given type of expertise and/or experiences to ensure all rationales are exposed Early experiments by Dalkey on rationales for specific subjective With N types needed for a complex problem this is 3xN to 5xN participants required Different backgrounds that require elimination of ambiguity and misinterpretations as well as translations of concepts Severe disagreements to be mediated Freedom of expression and improvisation of ideas without loss of face One out of ten ideas valuable High status participants most concerned about this Equal Participation allowed Minimize group process losses An idea can be brought up at anytime

12. © 2009 Turoff 12 Critical Success Factors Composition and quality of the participants Results only as good as the people involved Compensation for Effort Communication with a peer group Letting participants nominate others helps (snowball) Will learn from others in effort Results will serve useful and important purpose Consulting pay if above not strong enough What is obvious already included, participants used for what is not obvious (no blank sheets of paper) A morphological structure to automatically organize input material Important functions and objectives: exploration, understanding one another, exposure of issues and uncertainties, examination of disagreements, and generation of agreements when possible

13. © 2009 Turoff 13 Why Delphi: operational challenges Subjective judgments required Models to support consistency of judgments by individuals and group Building collective models (e.g. cross impact modeling, structural modeling) Producing collective Group views (e.g. Arrow’s paradox and scaling methods) Individual human biases (~10) Group process losses and gains (~20) Avoiding consensus (Asch & Hawthorne effects) pressure Participant effort and shadow time Asynchronous flexibility to use any time Language or cultural difficulties Virtual teams Detecting differences by backgrounds for feedback Multitasking, Cognitive Limitations, Information Overload

14. © 2009 Turoff 14 Delphi technology and methods Exploring Capturing individual knowledge Process of design, problem solving, derivation, knowledge structures Animation methods Promoting Understanding Forming a group synthesis Scaling methods Feedback to group Social judgment & voting theory Finalizing Viewpoints (no more changes) Evaluation by the group Collaborative processes

15. © 2009 Turoff 15 Feedback: a Thurstone Scale for Relative Importance of Measures of Disaster Impact Graduate Student class Delphi in an Emergency Management class with about 30% with real experience in the area.

16. © 2009 Turoff 16 Paired Comparison for Online Version

17. © 2009 Turoff 17 Thurstone Scale for Practice Problem System being built in Sahana open source effort for real time Decision analysis

18. © 2009 Turoff 18 Online Delphi’s Today Dynamic: entering main voting items, voting, making vote changes, and pro, con, neutral discussion comments, can all go on at once. Each individual can focus on what they want to System notifies user of changes Termination of vote changes is a sign of finalization of the results People not confident about a given item can always not vote or defer their vote to later People can propose rewording (Roberts Rules of Order) What took three months can happen in three weeks for complex problems and large groups Straight forward problems (e.g. emergency allocation priorities) can be done in real time by dispersed virtual groups of 5-20 professionals. Typically voting anonymous, comments signed Voting really serves to focus discussion on differences of view Used as a collaborative learning tool in online class discussions System designed as a Social Decision Support System or Dynamic Delphi

19. © 2009 Turoff 19 Approaches to the Future Goals affect what questions to ask Automation Reduce people time and effort Eliminate jobs Productivity Increase quantity Increase quality Opportunity Do new things Do things differently & Better

20. © 2009 Turoff 20 Approaches to the Future Dimensions of Group Communications Information Exchange Pooling of current data and information Cooperation Informing of plans and actions/efforts Coordination Joint planning of actions/efforts Collaboration Working together on the same actions/effort Too often above words are used as if they were all the same

21. © 2009 Turoff 21 Examples of general types of Delphi and Dynamic Delphis

22. © 2009 Turoff 22 Examples of General Delphi Designs and Dynamic Delphis These have all been done with paper a number of times Some have been done partially or totally online or in combination They could all be done online in a mode where anyone can work on any part at anytime Some of these still require a human monitor to perform certain functions and some can be fully automatic Users can: 1. Add new items or content 2. vote on any item, change their vote, and/or choose not to vote, and/or vote later after discussion 3. Suggest alternative wordings to existing item or add a new version of an existing item.

23. © 2009 Turoff 23 System operations Allow lists to which options, solutions, goals, etc can be added Allow voting scales for such lists Allow comments on list items Provide each user with new items of any type and also any whole discussion they want to see Provide vote summaries to indicated the relative status of voted items and highlight differences Determine any different vote patterns by participants with different expertise and/or experience, and/or backgrounds Show for any item Number voting Number of vote changes Number who might yet vote Number of discussion entries about an item

24. © 2009 Turoff 24 Delphi Examples Trend Delphi: produces a forecast of a trend along with the mental model of the group making the extrapolation of the trend curve into the future Problem Solving Delphi: Collects solutions to the problem which are rescaled to a group interval scale based upon individuals ranking or paired comparisons. Use voting to focus discussion on items that need it. Policy Delphi: seeks policy resolutions and the strongest pro and con evidence or arguments to support each policy resolution Cross Impact Modeling: Collaborative building of a model of the future possible outcomes of a set of unique events.

25. © 2009 Turoff 25 Trend Delphi System FunctionsParticipants ResponsesSystem Actions Present a historical trend to be extrapolated by the participants Draw a future curve or redraw a new one when a change has occurred in viewpoint. Present summary of 50% median and 0%, 25%, 75% and 100% boundaries Request assumptions and uncertainties used to make above estimate Turn all these into potential assumptions Vote on validity scale for each potential assumption. Scale is from completely true to completely false. Reorder assumptions from true to false. Focus on middle range (maybe) and ask which can be influenced or measured for occurrence Assume these can reduce the future uncertainty in the curve Ask for a redrawing of curve extrapolation based upon assumption list for each trend curve in the study Supply suggestions on how to influence or measure the maybe assumptions causing significant uncertainty in the projected curve. Summarize important findings at any time: Trend, true and false assumptions, assumptions that cause uncertainty, and their potential actions, and measurements

26. © 2009 Turoff 26 Problem Solving Delphi System FunctionsParticipants ResponsesSystem Actions State the problem and request solution options Provide options to solve the problem Present options in order of occurrence Request paired comparisons to measure individual preferences for options Make comparisons for option pairs that a participant feels confident about judging at any time. Use Thurstone's law of comparative judgment (using incomplete information) to derive a single group interval scale. Calculate uncertainty due to those who have not yet voted with same type of scale. Show interval scale. This indicates disagreements when two or more items are close together. This also shows clustering. Ask for comments about items where people disagree with current position of an item. Make comments about items you want to see others change their votes about. Present discussions about items for review. As more people vote or change votes scales will reflect decreasing uncertainty and often more separation between options.

27. © 2009 Turoff 27 Policy Delphi System FunctionsParticipants ResponsesSystem Actions State a policy issue to be examined. Ask for specific policy solutions Add resolution options or specific policies Request vote for Desirability and Feasibility scales of each solution Plot two dimensional distribution of policy resolutions Exploring desirable but infeasible solutions often important Request comments especially about those showing disagreement Request comments about policy resolutions. Indicate if comment is pro, con, or neutral. Request vote on comments for importance and validity It might be considered important because others believe it to be true A person may think a comment is important because others think it is valid. Do same two dimensional plots and summarize discussions

28. © 2009 Turoff 28 Cross Impact Modeling I System FunctionsParticipants ResponsesSystem Actions Use problem solving Delphi to produce a set of future unique events focused on a given situation Evaluate those events for their relative importance to the future objective guiding the choice of events Place the final most important events into a cross impact model Ask each individual to answer: What are the probabilities of each event occurring in some future time frame? Tell them for each event that they should assume it will or will not occur and ask them to express any changes in the probabilities of the other events due to that certain knowledge about the future. Show them the expected outcome of their model which will have differences from their predictions. Allow them to vary initial probabilities to see how the future changes. Allow them to go back and modify some of their estimates Create the cross impact model using the approach by Turoff (logistic, Fermi Dirac equations). This provides a scale changes from nonlinear probabilities (0 to 1) to a linear influence factor between each pair of events (plus to minus infinity). When participants are satisfied with their individual model utilize the internal linear influencing factors (Cij) to create a group model.

29. © 2009 Turoff 29 Cross Impact Modeling II Creating Scenarios System FunctionsParticipant ResponsesSystem Actions Analyze the internal parameters to show people which of the relationships between which events show the most disagreement among the group. Ask for comments on these combinations form those who have inconsistent or extreme views. Show these comments and others and allow those who wish to change some of their original estimations. Create a model of interacting scenarios by voting on where to stop the integration of the events in process that can turn all the events into one scenario When no more changes are being made produce. Use Interpretive Structural modeling to generate a set of macro scenarios collecting individual events that are tightly coupled into a set of scenarios that interact. Requires human monitor to know when to trigger the scenario creation part

30. © 2009 Turoff 30 Some appropriate words of wisdom

31. © 2009 Turoff 31 From “The people, yes” by Carl Sandburg The white man drew a small circle in the sand and told the red man, “This is what the Indian, knows,” and drawing a big circle around the small one, “This is what the white man knows.” The Indian took the stick and swept an immense ring around both circles: This is where the white man and the red man know nothing.” (The foresight paradox!) A seer upon perceiving a flood should be the first to climb a tree – Kahlil Gibran

32. © 2009 Turoff 32 Three slides on recommendations in higher education follow

33. © 2009 Turoff 33 Recommendation for Higher Education I Major effort to provide online discussion systems for all regular face to face courses Do not emphasize any integration with administrative services Avoid separate administration services just support of the hardware and software Seek ability to easily add communication learning features For example question & answer discussion where student cannot see ongoing answers and follow on discussion until they have answered the question Instructors must have control of setting up conferences and membership Ability to bring in guest lectures Ability to set up team conferences Ability to merge different classes for certain projects and exercises

34. © 2009 Turoff 34 Recommendation for Higher Education II Set up prizes and recognition for best online learning exercises in different areas Provide some guidance, seminars, and workshops for teaching online Provide some added equipment (cheap) for an instructor to able to record mp3 audio during his real class and be able to put it and slides online that evening With separate slides and audio students can view printout while listening to lecture off line and/or when traveling Allow those instructors willing to accept online students to his or her face-to-face class (so sick or traveling students do not have to miss anything) Allow face to face students to enroll in some online courses Set a goal of no difference between face-to-face and online student courses, instructors, and or requirements Offer prizes for student projects to develop improved learning tools for use in what ever is the system supporting the online discussion spaces

35. © 2009 Turoff 35 Recommendations for Higher Education III Allow universities to arranging cross registrations to share courses where a given campus does not offer useful courses Develop missing courses for local pre college schools that can be brought in online for areas where there are missing qualified teachers Don’t ignore adult education, literacy programs, and skill training programs and online distribution Consider at some point reducing face to face courses to 1/2 the time in class and put more emphasis on utilizing online time More discussion doable on line then ever can occur in most face-to-face classes. Start marketing internationally at some point Ask now what are the consequences of not moving forward in this direction as other countries begin to introduce online courses for Romania The time to lay out an action plan is now

36. © 2009 Turoff 36 The following slides contain references for those that want to learn more

37. © 2009 Turoff 37 Papers on my website Complete copy of original reference book on the Delphi Method The Delphi Method: Techniques and Applications by Harold Linstone and Murray Turoff, 1975. Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H., Yao, X., (2004),Online Collaborative Learning Enhancement through the Delphi Method, Proceedings of the OZCHI 2004 Conference, November 22-24, University of Wollongong, AustraliaOnline Collaborative Learning Enhancement through the Delphi Method Murray Turoff, Starr Roxanne Hiltz, Michael Bieber, Ajaz Rana (1998), Collaborative Discourse Structures in Computer Mediated Group Communications, 1998Collaborative Discourse Structures in Computer Mediated Group Communications Turoff, Murray, and Starr Roxanne Hiltz (1996), Computer Based Delphi Processes,a version will appear as an INVITED BOOK CHAPTER for Michael Adler and Erio Ziglio, editors., Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, London, Kingsley Publishers (in press).Computer Based Delphi Processes, Turoff, Murray (1997) Alternative Futures for Distance Learning: The Force and the Darkside, The material in this paper was utilized for an Invited Keynote Presentation at the UNESCO / OPEN UNIVERSITY International Colloquium, April 27-29: Virtual Learning Environments and the Role of the Teacher, Open University, Milton Keynes. It also forms the basis of a planned talk at the Third International ALN (Asynchronous Learning Networks) meeting in NY City, October, 1997. (Presentation Overheads for ALN conference, October, 1997, NYU)Alternative Futures for Distance Learning: The Force and the Darkside,Presentation Overheads Turoff, Murray (1995), Software Design and the Future of the Virtual Classroom, Journal of Information Technology for Teacher Education, Vol 4, No. 2, 1995Software Design and the Future of the Virtual Classroom Turoff, Murray (1999), An End to Student Segregation: No More Separation Between Distance Learning and Regular Courses. A summary of the invited plenary for the Telelearning 99 meeting in Montreal, Canada, November, 1999. (Also: ppt presentation used in talk.) An End to Student Segregation: No More Separation Between Distance Learning and Regular Courses.ppt presentation used in talk.

38. © 2009 Turoff 38 Other Papers 1 Turoff, M., Hiltz, S.R.: The Future of Professional Communities of Practice. In: Weinhardt, C., Luckner, S., Stößer, J. (eds.) WeB 2008. LNBIP, vol. 22, pp. 144-158. Springer-Verlag, Berlin Heidelberg (2009) Xiang, Y, Turoff, M., and Chumer, M. Designing a group support System to Review and Practice Emergency Plans in Virtual Teams, Proceedings of the 6th International ISCRAM Conference, Gothenburg, Sweden, May 2009 (http://iscram.org) White, Connie, Murray Turoff, Bartel Van de Walle, A Dynamic Delphi Process Utilizing a Modified Thurstone Scaling Method: Collaborative Judgment in Emergency Response, Proceedings of ISCRAM 2007, 4th International Conference on Information Systems for Crisis Response and Management, Delft, the Netherlands, May 13-16, Brussels University Press Plotnick, Linda, Elizabeth Avey Gomez, Connie White, Furthering Development of a Unified Emergency Scale Using Thurstone's Law of comparative Judgment: A progress Report, Proceedings of ISCRAM 2007, 4th International Conference on Information Systems for Crisis Response and Management, Delft, the Netherlands, May 13-16, Brussels University Press Turoff, Murray, Starr Roxanne Hiltz, Xiang Yao, Zheng Li, Yuanqiong Wang, and Hee-Kyung Cho, Online Collaborative Learning Enhancement Through the Delphi Method, Turkish Online Journal of Distance Education-TOJDE April 2006 ISSN 1302-6488 Volume: 7 Number: 2 Article: 6, Publisher: Anadolu University, Eskisehir, Turkey, http://tojde.anadolu.edu.tr/index.htm http://tojde.anadolu.edu.tr/index.htm Hee-Kyung Cho, Murray Turoff, and Starr Roxanne Hiltz, The Impact of Delphi Communication on Small and Medium Sized Asynchronous Groups: Preliminary Results, HICSS 36, January 2003, IEEE Computer Society Press. Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H. "The Delphi Process as a Collaborative Learning Method." In (edited by J. C. Moore) Elements of Quality Online Education: Into the Mainstream: Wisdom from the Sloan Consortium, 121-134. Needham, MA: Sloan-C, September 2004 Banuls, V., and Turoff, M., Scenario Construction via Cross-Impact, Draft under review 2009.

39. © 2009 Turoff 39 Other Papers 2 Cho, H.K. & Turoff, M., “Delphi Structure and Group Size in Asynchronous Computer-Mediated Communications,” Proceedings of the Americas Conference on Information Systems, Tampa, August 2003. Wang, Y., Li, Z., Turoff, M. and Hiltz, S.R. (2003). Using a social decision support system toolkit to evaluate achieved course objectives. Proceedings of the Americas Conference on Information Systems, Tampa, August. (Nominated as a “best paper.”) Turoff, Murray and S. R. Hiltz, (1995), Computer Based Delphi Processes, in Michael Adler and Erio Ziglio, editors., Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, London, Kingsley Publishers, pp. 56-88. Worrell, W., Hiltz, S. R., Turoff, M. and Fjermestad, J. (1995) An experiment in collaborative learning using a game and a computer-mediated conference in accounting games. Proceedings of the 28th Annual Hawaii International Conference on System Sciences, Vol. IV, pp. 63-71. Los Alamitos, CA: IEEE Computer Society Press, 1995. Hsu, Enrico Y. P., Hiltz, S. R., and Turoff, Murray (1992). Computer-Mediated Conferencing System as Applied to a Business Curriculum: A Research Update. In V. S. Jacob and H. Pirkul, eds., The Impact of Information Technology on Business Schools: Research, Teaching and Administration, Proceedings of the 20th Annual North American Conference of the International Business School Computer Users Group, pp. 214- 227. Awarded "Best Paper- Teaching.“ Hiltz, S.R. and Turoff, M., The Network Nation: Human Communication via Computer, 1978, revised edition reprinted 1993 by MIT Press