1. Tools for Assessing Perceptions and Uncovering Influence Jim Dearing Center for Health Dissemination and Implementation Research Kaiser Permanente Colorado Synergy Project for Research, Practice and Transformation January 10-12, 2010, Albuquerque NM

2. 2 Session Objective: To help you understand methods of data collection and measures for the assessment of perception and influence

3. Formative evaluation is a type of applied research that is conducted prior to the introduction of an innovation to increase the likelihood of achieving scale 3

4. Formative Evaluation of Two Types 1. Learning about the innovation 2. Learning about potential adopters 4

5. Learning about the Innovation  Is based in the assessment of perception Whose perception most counts? Whose perception most counts? From what other types of stakeholders might we want to gather data about perception? From what other types of stakeholders might we want to gather data about perception? 5

6. Learning about the Innovation  How can we learn of perceptions?  Two methods of data-collection Interviews with open-ended response categories Interviews with open-ended response categories Questionnaires with closed-ended response categories Questionnaires with closed-ended response categories  Under what conditions might we prefer to collect data by interview vs. questionnaire? 6

7. 7 Interviews are Preferable When  Interviewees are very high-ranking, few in number, when the topic is especially sensitive, or when we have reason to doubt that standard attributes will well- represent the characteristics of the innovation  Always pretest your interview protocol  Is this human subjects research? IRB IRB

8. Collection of Data via Interviews has Its Downside  Real-time jotting down of comments 2 nd person? 2 nd person?  Digital recording Transcription, training, coding into attribute categories, organizing of data, analysis of frequencies, reporting Transcription, training, coding into attribute categories, organizing of data, analysis of frequencies, reporting 8

9. A Questionnaire can be Brief 1. Compatibility 2. Cost 3. Simplicity 4. Adaptability 5. Effectiveness 6. Observability 7. Trialability 9

10. Simple Tallies of Responses can be Insightful  The more respondents the better staff perception, staff portrayals, potential adopter perception staff perception, staff portrayals, potential adopter perception which of these three types of respondents is most important? which of these three types of respondents is most important?  Use mean scores per attribute Whether for creating attribute matrices or going straight to an attribute profile (1x7 data representation per innovation) Whether for creating attribute matrices or going straight to an attribute profile (1x7 data representation per innovation) 10

11. Tools for Assessing and Comparing Innovations   Attribute matrix, innovation profile, and potential for adoption (PAR) score are simple ways of organizing numerical data purpose is standardization, diagnosis, improvement, comparison Dearing JW, Meyer G (1994). An exploratory tool for predicting adoption decisions. Science Communication 16(1):43-57.

12. Discussion  Do you have one priority innovation that you’re considering for scale-up? what sorts of feedback have you gotten to- date about it, and from what type of stakeholders? what sorts of feedback have you gotten to- date about it, and from what type of stakeholders?  How do you decide who to collect formative evaluation data from? Do you sample from the target population and if so, how? Do you sample from the target population and if so, how? 12

13. Does All This Suggest Too Much Rigor?  Formative evaluation is usually done with small numbers of people (even questionnaires)  These means of learning about the innovation are meant to supplement other means of helping you decide which innovations are ready  You don’t need to overcome bias but you do want to understand it 13

14. Learning about Potential Adopters   Data-collection can be done in one of four ways: informant interviews observation sociometric survey self-report All for assessing social influence

15. What Difference Does It Make if We Work with Influentials?  Look at the degree of “reach” that people have in influence networks 15

16. A typical KP Colorado employee’s 2-step network neighborhood

17. An established KP Colorado employee’s 2-step network neighborhood

18. A KP Colorado bridging individual’s 2-step network neighborhood

19. Learning about Potential Adopters   Informant interviews (snowball process) suited for large social systems identify a few informants with broad knowledge of many others (1 st round interviews) ask their opinion of whom among potential adopters they think are looked to by others (2 nd round interviews, beginning of data-collection) 3 rd round interviews, 4 th round interviews, etc stop when no new names are being generated

20. Learning about Potential Adopters   Observation suited to small social systems train knowledgeable observers to recognize the behaviors associated with social influence what are these? recording sheets (roster format)

21. Learning about Potential Adopters   Sociometric survey who-to-whom questions   Well-suited to medium size social systems   Simple administration, simple data-entry   Wholly different form of relational data as a result   Best method for identifying and representing influence   Partner with a grad student?

22. Learning about Potential Adopters   Self-report survey large social systems (marketing research) lower inherent validity but validated instruments still available less intrusive more typical types of questions   Partner with a grad student?

23. Working Through Opinion Leaders   Does intervention with opinion leaders work to speed and spread the scale-up of innovations? Yes, generally efficacious Depends on how you recruit them Depends on what you ask them to do

24. Recruitment & the Request  Your appeal should be normative in nature  Ask them to continue to act with the best interests of the network in mind  To evaluate the pros and cons of innovations communicated to them  To talk, to refer, to suggest, all within their own range of everyday behaviors 24

25. In Sum  I have married an outstanding woman. I owe all of my accomplishments to her – she is brilliant, patient, orderly and neat, and sexy beyond words. And as a mother, she sets the example for all professional, hard working women. 25

26. In Sum   Formative data-collection can be conducted to improve the likelihood that your innovation can reach more potential adopters, more rapidly, and produce positive perceptions among them a range of tools exist for measurement of both personal perceptions and social influence

27. 27 www.research-practice.org james.w.dearing@kp.org Dearing JW (2009). Applying diffusion of innovation theory to intervention development, Research on Social Work Practice 19: 503-518. james.w.dearing@kp.org