1. Making a Difference Through Program Evaluation Michael Quinn Patton September 21, 2010 Program Evaluation Quality Assurance Summit 1

2. LBJ 2

3. 3 Globalization of the evaluation community – of which you are all a part.

4. Burkina Faso 4

5. Evaluation as a transdiscipline and a profession… with quality assurance Standards and professional development opportunities. 5

6. Standards – Quality Assurance  Utility – ensure relevance & use  Feasibility – realistic, prudent, diplomatic & frugal  Propriety – ethical, legal, respectful  Accuracy – technically adequate to determine merit or worth For the full list of Standards: www.wmich.edu/evalctr/checklists/standardschecklist.htm 6

7. Meta-evaluation Evaluating evaluations for ongoing learning, improvement and quality assurance 7

8. Quality Assurance Ensuring Value through Evaluation E-VALU-ate 8

9. Quality Assurance Quality control and/or Quality enhancement 9

10. Personal Factor: Intended Use by Intended Users 10

11. Critical success factors: There are five key variables that are absolutely critical in evaluation use. They are, in order of importance: People – People People – People » PEOPLE 11

12. Big news Evaluators are people too 12

13. Basic premise Value and quality assurance in Evaluation comes through EVALUATORS 13

14. Essential skills Add value by being good at what you do and knowing your role and playing it well 14

15. Know thyself Connais-toi toi-même Greek: γνῶθι σεαυτόν Latin: nosce te ipsum 15

16. Essential Competencies for Program Evaluators Jean King, a recipient of the American Evaluation Association’s prestigious Alva and Gunnar Myrdal Practice Award, has worked for a number of years with colleagues and students conducting research on and developing a framework for Essential Competencies for Program Evaluators (Ghere et al. 2006; King et al., 2001).  The final product is a taxonomy of essential program evaluator competencies organized into six primary categories. 16

17. Essential Competencies 1. Professional Practice: knowing and observing professional norms and values, including evaluation standards and principles. 2. Systematic Inquiry: expertise in the technical aspects of evaluations, such as design, measurement, data analysis, interpretation, and sharing results. 3. Situational Analysis: understanding and attending to the contextual and political issues of an evaluation, including determining evaluabiity, addressing conflicts, and attending to issues of evaluation use. 17

18. Essential Competencies 4. Project Management: the nuts and bolts of managing an evaluation from beginning to end, including negotiating contracts, budgeting, identifying and coordinating needed resources, and conducting the evaluation in a timely manner. 5. Reflective Practice: an awareness of one’s program evaluation expertise as well as the needs for professional growth. 18

19. Essential Competencies 6. Interpersonal Competence: the people skills needed to work with diverse groups of stakeholders to conduct program evaluations, including written and oral communication, negotiation, and cross-cultural skills. 19

20. Essential skills Add value by being good at what you do and knowing your role and playing it well 20

21. The Nature of Expertise and the role of expert 21

22. What’s it take to achieve world class expertise? 10,000 hours 22

23. Identities we take on Evaluator Methodologist Researcher Auditor Inspector Learning facilitator Judge These roles are necessary, important and useful, but there’s another role that offers value: The jester 23

24. 24

25. SPEAKING TRUTH TO POWER 25

26. Jesting is Serious Business because Speaking TRUTH to Power is serious business 26

27. In the beginning… 27

28. 28

29. Aboutthis bookAboutthis book Preview this bookPreview this book Shake Hands with the Devil By Roméo DallaireShake Hands with the Devil By Roméo Dallaire This is a preview. The total pages displayed will be limited. LL Loading... Loading... 29

30. Value-added Challenge: Matching the evaluation process and design to the nature of the situation to achieve intended use by intended users: Contingency-based Evaluation 30 to achiev e intend ed use by intend ed users:

31. Evaluation’s General, Generic Value Value of a Particular Evaluation and Specific Approach 31

32. Mrs. McCave and her 23 Daves 32

33. Not just experience but Focused Practice and REFLECTION on experience to achieve situational responsiveness 33

34. Story about ongoing learning 34

35. 35

36. Executing Doing what we know how to do and know we should do 36

37. The Fine Art of Selling Successful Selling Methods What Makes a Great Salesperson? The Secret to Selling Increasing Sales Now Sales Tips from Great Salesmen Topping the Sales Charts Breakthrough Strategies For Selling to Difficult People Expert Hands-On Sales Techniques Improving Sales Performance - Where to Begin? Sales Techniques Revealed 37

38. Lessons from Sales Experts 1.Believe in your product 2.Know your product 3.Connect your product to what your customer needs: listen communicate connect 38

39. Elevator speech test Can you say clearly, in 30 seconds what your product is, why you believe in, and say it in a way that connects with the person you’re talking to? 39 Elevator test

40. Believe in your product 40

41. Bob Stake, “Beyond Neutrality: What Evaluators Care About” 41

42. Beyond Neutrality: What Evaluators Care About 1. We often care about the thing being evaluated. 2. We, as evaluation professionals, care about evaluation. 3. We advocate rationality. 4. We care to be heard. We are troubled if our studies are not used. 5. We are distressed by underprivilege. We see gaps among privileged patrons and managers and staff and underprivileged participants and communities. 6. We are advocates of a democratic society. SOURCE: Robert Stake (2004:103–107). 42

43. Know your product 43

44. Value-added Challenge: Matching the evaluation process and design to the nature of the situation to achieve intended use by intended users Contingency-based Evaluation 44

45. Connect your product to what your customer needs: listen communicate connect 45

46. 46

47. 47

48. PROCESS USE 48

49. Three Cups of Tea Baltistani proverb: First cup you share, you are a stranger. Second cup, you are an honored guest. Third cup, you are in relationship. 49

50. Enhancing Quality in Evaluation 1.Believe in evaluation 2.Know evaluation (and yourself as evaluator) 3.Connect to your primary intended users: listen communicate connect 50

51. 51

52. 52