Jeanette Gurrola Psychology Department School of Behavioral & Organizational Sciences Claremont Graduate University American Evaluation Association Conference 2010

The American Evaluation Association is formed Kirkhart: Defining evaluator competencies: New light on an old issue Program Evaluation Standards are published 1995 Guiding Principles for Evaluators are published 1996 Scriven: Types of evaluation and types of evaluator 2001 King et al.: Toward a taxonomy of essential evaluator competencies 2005 Stevahn et al.: Establishing essential competencies for program evaluators 2009 CES: Competencies for Canadian evaluation practice

Kirkhart (1981) Scriven (1996) King et al. (2001) Stevahn et al. (2005) CES (2009) Empirically derivedXXX Competency Domains Interpersonal practiceXXXX Professional ethicsXXXX Project managementXXXX Reflective practice XX Situational analysisXXXXX Systematic inquiryXXXXX

 Do evaluation approaches emphasize that evaluators be competent in certain skills and knowledge?  Additionally, do evaluation approaches vary on which evaluator competencies they emphasize?

Christie & Alkin (2008) Evaluation Theory Tree Donaldson

 Donaldson ◦ Methods branch ◦ Theory-driven program evaluation  Greene ◦ Values branch ◦ Responsive, inclusive, mixed methodology  King ◦ Use branch ◦ Evaluation capacity building

 Refereed journal articles  Published in the past ten years  Practical application of an evaluation  Content analysis  Stevahn et al. (2005) ECPEs ◦ 61 competencies across 6 domains ◦ Rated degrees of referenced  Not referenced, referenced, extensively referenced  Descriptive and Chi-square analyses

 Professional Practice  Systematic Inquiry  Situational Analysis  Project Management  Reflective Practice  Interpersonal Competence

 Professional Practice  Systematic Inquiry  Situational Analysis  Project Management  Reflective Practice  Interpersonal Competence Applies professional evaluation standards Acts ethically and strives for integrity and honesty in conducting evaluations Conveys Personal evaluation approaches and skills to potential clients Respects clients, respondents, program participants, and other stakeholders Considers the general and public welfare in evaluation practice Contributes to the knowledge base of evaluation

 Professional Practice  Systematic Inquiry  Situational Analysis  Project Management  Reflective Practice  Interpersonal Competence Understands the knowledge base of evaluation Knowledgeable about methodology Conducts literature reviews Specifies program theory Frames evaluation questions Develops evaluation designs Identifies data sources Collects data Assesses validity and reliability of data Analyzes data Interprets data Makes judgments Develops recommendations Provides rationales for decisions throughout the evaluation Reports evaluation procedures and results Notes strengths and limitations of the evaluation Conducts meta-evaluations

 Professional Practice  Systematic Inquiry  Situational Analysis  Project Management  Reflective Practice  Interpersonal Competence Describes the program Determines program evaluability Identifies the interests of relevant stakeholders Serves the information needs of intended users Addresses conflict Examines the organizational context of the evaluation Analyzes the political consideration relevant to the evaluation Attends to the issues of evaluation use Attends to the issues of organizational change Respects the uniqueness of the evaluation site and client Remains open to input from others Modifies the study as needed

 Professional Practice  Systematic Inquiry  Situational Analysis  Project Management  Reflective Practice  Interpersonal Competence Responds to request for proposals Negotiates with clients before the evaluation begins Writes formal agreements Communicates with clients throughout the evaluation process Budgets an evaluation Justifies costs given information needs Identifies needed resources for evaluation Uses appropriate technology Supervises others involved in conducting the evaluation Trains others involved in conducting the evaluation Conducts the evaluation in a nondisruptive manner Presents work in a timely manner

 Professional Practice  Systematic Inquiry  Situational Analysis  Project Management  Reflective Practice  Interpersonal Competence Aware of self as an evaluator Reflects on personal evaluation practice Pursues professional development in evaluation Pursues professional development in relevant content areas Builds professional relationships to enhance evaluation practice

 Professional Practice  Systematic Inquiry  Situational Analysis  Project Management  Reflective Practice  Interpersonal Competence Uses written communication skills Uses verbal/listening communication skills Uses negotiation skills Uses conflict resolution skills Facilitates constructive interpersonal interaction Demonstrates cross-cultural competence

Competency DomainsAbsentPresent Professional Practice 67%33% Systematic Inquiry 50%50% 2 Situational Analysis 77%23% Project Management 75%25% Reflective Practice 90%10% Interpersonal Competence 67%33% Total 67%33% a Pearson X 2 statistically significant at.05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage.

Competency DomainsAbsent Referenced Extensive Professional Practice 67%25%8% Systematic Inquiry 50%19%31% Situational Analysis 77%4%19% Project Management 75%19%6% Reflective Practice 90%5% Interpersonal Competence 67%4%29% Total 67%14%19% a Pearson X 2 statistically significant at.05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage. Degrees of Referenced

Competency Domains % Extensive Systematic Inquiry31% Interpersonal Competence29% Situational Analysis19% Professional Practice8% Project Management6% Reflective Practice5%

Competency Domains % Extensive Systematic Inquiry31% Interpersonal Competence29% Situational Analysis19% Professional Practice8% Project Management6% Reflective Practice5% (4) Collects data (2) Knowledgeable about qualitative methods (2) Conducts literature reviews (2) Specifies program theory (2) Frames evaluation questions (2) Develops evaluation design (2) Identifies data sources (2) Assesses validity of data (2) Assesses reliability of data (2) Analyzes data Specific Competencies (4 articles)

Competency Domains % Extensive Systematic Inquiry31% Interpersonal Competence29% Situational Analysis19% Professional Practice8% Project Management6% Reflective Practice5% (2) Uses verbal/listening communication skills (2) Demonstrates cross-cultural competence Specific Competencies (4 articles)

Competency DomainsAbsentPresent Professional Practice 71%29% Systematic Inquiry 58%42% Situational Analysis 38% 1 62% 2 Project Management 62%38% Reflective Practice 89% 2 11% 1 Interpersonal Competence 60%41% Total 59%41% a Pearson X 2 statistically significant at.05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage.

Competency DomainsAbsent Referenced Extensive Professional Practice 71%17%12% Systematic Inquiry 58%14%29% Situational Analysis 38% 1 21%41% 2 Project Management 62%30% 2 8% 1 Reflective Practice 89% 2 9%3% 1 Interpersonal Competence 60%31%10% Total 59%20%21% a Pearson X 2 statistically significant at.05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage. Degrees of Referenced

Competency Domains % Extensive Situational Analysis41% Systematic Inquiry29% Professional Practice12% Interpersonal Competence10% Project Management8% Reflective Practice3%

Competency Domains % Extensive Situational Analysis41% Systematic Inquiry29% Professional Practice12% Interpersonal Competence10% Project Management8% Reflective Practice3% (5) Describes the program (5) Serves the information needs of intended users (4) Identifies the interest of relevant stakeholders (4) Attends to the issues of evaluation use Specific Competencies (7 articles)

Competency Domains % Extensive Situational Analysis41% Systematic Inquiry29% Professional Practice12% Interpersonal Competence10% Project Management8% Reflective Practice3% (7) Collects data (6) Knowledgeable about quantitative methods (5) Reports evaluation procedures and results Specific Competencies (7 articles)

Competency DomainsAbsentPresent Professional Practice 75% 2 25% 1 Systematic Inquiry 38%63% Situational Analysis 21% 1 79% Project Management 50% Reflective Practice 45%55% Interpersonal Competence 21%79% Total 39%61% a Pearson X 2 statistically significant at.05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage.

Competency DomainsAbsent Referenced Extensive Professional Practice 75% 2 8%17% Systematic Inquiry 38% 25% Situational Analysis 21% 1 21%58% 2 Project Management 50%27%23% Reflective Practice 45%30%25% Interpersonal Competence 21%50%29% Total 39%30%31% a Pearson X 2 statistically significant at.05 level. b More than 25% of cells have an expected count less than 5. 1 Observed percentage statistically significantly less than expected percentage. 2 Observed percentage statistically significantly greater than expected percentage. Degrees of Referenced

Competency Domains % Extensive Situational Analysis58% Interpersonal Competence29% Systematic Inquiry25% Reflective Practice25% Project Management23% Professional Practice17%

Competency Domains % Extensive Situational Analysis58% Interpersonal Competence29% Systematic Inquiry25% Reflective Practice25% Project Management23% Professional Practice17% (4) Identifies the interest of relevant stakeholders (4) Attends to the issues of evaluation use (4) Remains open to input from others (3) Determines program evaluability (3) Addresses conflict (2) Describes the program (2) Examines the organizational context of the evaluation (2) Attends to the issues of organizational change Specific Competencies (4 articles)

Competency Domains % Extensive Situational Analysis58% Interpersonal Competence29% Systematic Inquiry25% Reflective Practice25% Project Management23% Professional Practice17% (3) Facilitates constructive interpersonal interaction (2) Demonstrates cross-cultural competence Specific Competencies (4 articles)

1 SI IC SA IC SI

 Donaldson ◦ Theory-driven program evaluation  Emphasis on Systematic Inquiry  Greene ◦ Responsive, inclusive, mixed methods approach  Emphasis on Situational Analysis  Lack of emphasis on Reflective Practice  King ◦ Evaluation capacity building  Emphasis on Situational Analysis  Lack of emphasis on Professional Practice

 Representative to only the three theorists  Articles not written with competencies in mind  Focus on specific competencies  Inform training, professional development, education, etc.  Contributes to research on evaluation  Future research needed to expand findings on evaluation approaches

 Canadian Evaluation Society (2009, May 31). Companion Document for Competencies for Canadian Evaluation Practice. Retrieved from  Christie, C.A. & Alkin, M. C. (2008). Evaluation theory tree re-examined. Studies in Educational Evaluation, 34,  King, J. A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22 (2),  Kirkhart, K. (1981). Defining evaluator competencies: New light on an old issue. American Journal of Evaluation, 2,  Perrin, B. (2005). How can information about the competencies required for evaluation be useful? The Canadian Journal of Program Evaluation, 20 (2),  McGuire, M. & Zorzi, R. (2005). Evaluator competencies and performance development. The Canadian Journal of Program Evaluation, 20 (2),  Scriven, M. (1996). Types of evaluation and types of evaluator. American Journal of Evaluation, 17 (2),  Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26 (1),

Jeanette Gurrola Psychology Department School of Behavioral & Organizational Sciences Claremont Graduate University