1. Essential Competencies for Program Evaluators Jean A. King 2012 AEA/CDC Summer Institute

2. Agenda for the session Identifying competencies intuitively Beliefs about evaluator competencies The ECPE and how they work in different settings Case application Do-it-yourself concept maps Self-assessment and planning for professional development

3. IDENTIFYING COMPETENCIES People create these things!

4. Identifying competencies intuitively What are the knowledge, skills and dispositions that a program evaluator needs to successfully complete the evaluation?

5. Evaluator relationships A relationship exists among –The evaluator –The client –Program staff and other evaluation stakeholders Primary responsibility for two tasks: –Making evaluation decisions –Implementing the evaluation

6. Evaluator relationships Primary responsibility for making and implementing evaluation decisions may lie (1) With the evaluator (2) With someone else in the setting (3) Jointly with both The evaluation relationship may shift during the study

7. Interactive participation quotient (IPQ) LOW HIGH Evaluator Program leaders, staff, community Involvement in decision making and implementation Participant- directed CollaborativeEvaluator- directed ZONES

8. COMPETENCY STATEMENTS What do you believe?

9. Question: To what extent do your personal beliefs matter in a discussion of evaluator competencies?

10. ESSENTIAL COMPETENCIES FOR PROGRAM EVALUATORS Presenting...

11. How were the ECPE developed? Began: With an idea in a graduate course Then: “What does the literature tell us?” Then: A pilot study using the initial set of competencies, AJE publication (2001) Then: A revision process including a crosswalk Then: ECPE published in AJE (2005) Then: Articles in CJPE (2005) and AJE (2006) Then: CES’s professional designation program Next: A national validation study

13. 1) Professional practice  Fundamental norms and values of evaluation practice - Applies evaluation standards - Works ethically - Respects all stakeholders - Considers the general and public welfare - Contributes to the knowledge base

14. 2) Systematic inquiry  Technical aspects of evaluation practice  Understands knowledge base  Knows quantitative and qualitative methods  Knows mixed methods  Conducts literature reviews  Specifies program theory  Frames evaluation questions  Designs the evaluation PLUS...

15. 2) Systematic inquiry (cont.)  Collects and interprets data  Makes judgments and recommendations  Provides rationales for decisions  Reports procedures and results  Notes evaluation strengths and limitations  Conducts meta-evaluations

16. 3) Situational analysis  Unique interests, issues, and contextual circumstances of evaluation  Describes program  Determines program evaluability  Identifies stakeholders’ interests  Serves intended users’ informational needs  Addresses conflicts  Attends to evaluation use PLUS...

17. 3) Situational analysis (cont.)  Examines organizational and political context  Attends to organizational change  Respects site and client uniquenesses  Remains open to input  Modifies evaluation as needed

18. 4) Project management  “Nuts and bolts” of evaluation work  Responds to RFPs  Negotiates with clients  Writes formal agreements  Communicates with client  Budgets evaluation  Justifies costPLUS...

19. 4) Project management (cont.)  Identifies needed resources  Uses technology as appropriate  Supervises and trains others  Conducts evaluation in a nondisruptive manner  Presents work in timely manner

20. 5) Reflective practice  One’s own evaluation expertise and need for growth - Knows self - Reflects on practice - Pursues professional development: evaluation and content areas - Builds professional relationships

21. 6) Interpersonal competence  “People skills” necessary for evaluation practice - Uses written and verbal/listening communication skills - Uses negotiation skills - Uses conflict resolution skills - Facilitates constructive interpersonal interaction - Demonstrates cross-cultural competence

22. AN INITIAL STUDY TO DEVELOP INSTRUMENTS AND VALIDATE THE ECPE A recent research study

23. Wilcox (2012) research study Top- and lowest-rated competencies (rated on a scale of 1 [not at all necessary] to 5 [extremely necessary]) Subject area differences not as important as evaluator role Results of 9 expert interviews

24. Top weighted competencies (Wilcox, 2012) LabelCompetency Weighted Mean 1.2 Acts ethically and strives for integrity and honesty in conducting evaluations 4.87 1.4 Respects clients, respondents, program participants, and other stakeholders 4.76 6.2 Uses verbal/listening communication skills 4.67 2.14 Interprets data 4.63 6.1 Uses written communication skills 4.55

25. Lowest weighted competencies (Wilcox, 2012) LabelCompetency Weighted Mean 2.5 Conducts literature reviews 3.61 2.6 Specifies program theory 3.55 4.1 Responds to requests for proposals 3.20 1.6 Contributes to the knowledge base of evaluation 3.09 2.20 Conducts meta-evaluations 2.70

26. Lowest weighted competencies (Wilcox, 2012) LabelCompetency Weight ed Mean 2.5 Conducts literature reviews 3.61 2.6 Specifies program theory 3.55 4.1 Responds to requests for proposals 3.20 1.6 Contributes to the knowledge base of evaluation 3.09 2.20 Conducts meta-evaluations2.70

27. Original and final interviewee categories OriginalFinal Education evaluators Public health evaluatorsEvaluators in formal organizations Evaluators of non-profitsEvaluations as independent consultants

28. Summary of expert interviews (Wilcox, 2012) ECPE Category In Education In Formal Organizations As Independent Consultants Professional practice Primary Primary Primary Systematic inquiry Essential [Reliance on assessment data] Essential [Team building] Essential [Collaboration] Interpersonal competence Very important Very important Very important

29. Summary of expert interviews (Wilcox, 2012) ECPE Category In Education In Formal Organizations As Independent Consultants Situational analysis Seldom Sometimes Frequently Project management Not so much Senior staff’s responsibility All the time Reflective practice Wish to be able to do more regularly Could do better Trying to keep up

30. Applying the competencies... 1.Back to the IPQ for discussion/reflection 2.A case application

31. ECPE CONCEPT MAPPING Now the fun part...

32. Work context and program evaluator competencies

33. Create a concept map-- In your work context, what are the relation- ships between and among the competencies?

34. SELF-ASSESSMENT What’s next for you?

35. ECPE self-assessment Reflecting on your own practice, how do you rate your current knowledge, skills, and dispositions to effectively carry out program evaluations?

36. ECPE rating scale Entry / Novice Proficient/ Skilled Mastery/ Expert

37. Now what? What is your professional development plan?

38. References Ghere, G., King, J.A., Stevahn, L., & Minnema, J. (2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27, 108-123. Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43-59.

39. Feel free to contact me at kingx004@umn.edu or see www.evaluation.umn.edu