1. Janet Maher

2.  Much of the project work is focused around the design, delivery, and testing of training materials for a range of audiences  We can draw on some experience for specific audiences but few models to accommodate the heterogeneity of audiences at hand  Limited time and resources to complete the project

3.  To devise a methodology and protocol for high quality collaborative decision making in the design, development and testing of training content

4.  What is it? ◦ a method for structuring a group communication process to allow effective processing of a complex problem (Linstone & Turoff, 1975)  Where does it come from? ◦ Project Delphi was a USAF operation in the 1950s focused on figuring out Soviet thinking at the height of the cold war—where a lot of existing experts varied widely in their estimates and conclusions

5.  How does it work? ◦ A panel of judges (experts) work independently and communicate through a moderator  Stage 1: a brainstorm of ideas or concepts—e.g. open- ended survey [results fed back to panel by moderator]  Stage 2: exploring the ideas and links between them— narrower survey [results fed back with further probes]  Stage 3: decision-making and prioritizing among them  (Stage 4: further refinement of the decision/options, can also add one or more face to face sessions)

6.  What are some of the issues? ◦ Selecting the panel of experts—heterogeneity and who counts ◦ Optimal number of experts--how representative and how valid will the results be ◦ Reducing bias—lots of room for ‘manipulation’ or ‘subversion’ of the process Some of this can be addressed by formalizing the process in a protocol or guidelins

7. 1.Preparation  Experience  Literature review  Pilot or previous studies 2.Research Question  Research Design  Sample 3.Delphi Round 1 Design  Pilot  Survey & analysis  Feedback for round 2

8. 4. Delphi Round 2 Design  Survey & Analysis  Feedback for round 3 5. Delphi Round 3 Design  Survey & Analysis  Summary and presentation 6. Research documentation, verification and generalization

9.  Provides for a comprehensive documentation of the process—as part of the formative evaluation  Can engage SMEs in a limited but predictable and important way in project design and development  Can accommodate heterogeneity of audiences (and presumably range of expertise)

10.  Have we got the right/best experts?  Do we agree on scoring and rules used by moderator?  Do we have a process for dealing with variability in responsiveness of experts?  Do we have a process for feedback checking/confirming?

11.  Do we want to pursue a guideline/protocol?  If so, which parts of the content development process should be included  Thanks  Janet