Factors Impeding and Enhancing Exemplary Evaluation Practice Nick L. Smith Syracuse University Presentation at the American Evaluation Association annual meeting, Anaheim, CA, November, 2011.

Overview Report on ongoing line of inquiry on exemplary evaluation practice. Began with AJE Exemplars Section, Paul R. Brandon lead co-editor. Descriptive approach investigating the nature of exemplary evaluation practice. Third in a series of reports.

The Problem Few studies are recognized, many more are likely exemplary.  Awards to large, well-funded, highly visible evaluations, more modest efforts overlooked. Individual, external studies recognized.  Also exemplary: multiple, targeted studies; internal systemic evaluations; consulting relationships; meta- evaluations. Design and/or impact are exemplary.  Other evaluation aspects may be exemplary.

The Problem Limited systematic study of exemplariness.  Awards based on expert panel review, implicit criteria, self-promotion, or general acclaim.  Reports of exemplary practice often based on evaluator self-report.  Few critical, comparative investigations of exemplary work. Utility of answers.  Improve practice, evaluator training, and theory- practice interface.

Preliminary Observations Selected issues under consideration.  What constitutes exemplariness?  What aspects of an evaluation can be exemplary?  Does what is considered exemplary differ by context of work?  Are certain conditions necessary for exemplary practice?  Does what is considered exemplary differ by type of evaluation work?  What factors impede or enhance exemplary work?  Who determines what is exemplary? Evaluator? Client? Society? Profession?

American Evaluation Association 2010 Smith, N. L. & Tolley, L. M. Exemplary evaluations: Characteristics, conditions and considerations, presentation at the American Evaluation Association annual meeting, San Antonio, TX, November, 2010.

Information Sources Prior Cases: Fitzpatrick, J., Christie, C., & Mark, M. M. (2009). Evaluation in action: Interviews with expert evaluators. Los Angeles: Sage.  Ricco, J., The Evaluation of GAIN: A Welfare-to-Work Program in California.  Greene, J., Evaluation of the Natural Resources Leadership Program, 1995 Through  Bickman, L., The Evaluation of the Ft. Bragg and Stark County Systems of Care for Children and Adolescents.  Fetterman, D., The Evaluation of the Stanford Teacher Education Program (STEP).

Information Sources Prior Cases: Fitzpatrick, J., Christie, C., & Mark, M. M. (2009). Evaluation in action: Interviews with expert evaluators. Los Angeles: Sage.  Rog, D., The Evaluation of the Homeless FamiliesProgram.  Henry, G., The Council for School Performance: Performance Reports for Georgia Schools.  King, J., Evaluation of the Special Education Program at the Anoka-Hennepin School District.  Donaldson, S., Evaluation of the Work and Health Initiative With a Focus on Winning New Jobs.

Information Sources Prior Cases: Fitzpatrick, J., Christie, C., & Mark, M. M. (2009). Evaluation in action: Interviews with expert evaluators. Los Angeles: Sage.  Preskill, H., Developing an Evaluation System for the Corporate Education Development and Training (CEDT) Department at Sandia National Laboratories.  Conner, R., Evaluation of the Colorado Healthy Communities Initiative.  Bledsoe, K., Evaluation of the Fun With Books Program.  Wallis, A. & Dukay, V., Evaluation of Godfrey’s Children Center in Tanzania.

Preliminary Observations What aspects of an evaluation can be exemplary? Strong Study Design* Good Implementation Effective Resource Management Responsiveness to Info Needs of Clients and Stakeholders Timeliness with respect to Decisions, Actions, and Conditions Context and Culturally Sensitive Process and Results are Politically Credible, Acceptable, Useable Results are Relevant, Feasible, Usable, and Accessible Evaluator Openness and Flexibility to Changing Conditions, Client Interests/needs, Available Resources, etc. Planning for and Implementation of Subsequent Use of Evaluation Results and Process

Preliminary Observations Does what is considered exemplary differ by context of work?  Is exemplary evaluation work different depending on whether one works in government, business, health, education, the arts, military, environmental studies?  A for-profit business might most value an evaluation that promotes a market-responsive learning organization.  A government agency might value an evaluation that preserves agency independence while also ensuring public accountability (Chelimsky, 2010).

Eastern Evaluation Research Society 2011 Smith, N. L. The nature of exemplary evaluation work: The evaluator-client relationship, presentation at the annual meeting of the Eastern Evaluation Research Society, Galloway, NJ, May, 2011.

AJE Collaborative Analysis  Smith, N. L., Brandon, P. R., Lawton, B. E., & Krohn- Ching, V. (2010). Evaluation exemplar: Exemplary aspects of a small group-randomized local educational program evaluation. American Journal of Evaluation, 31(2),  Brandon, P. R., Smith, N. L., Trenholm, C., & Devaney, B. (2010). Evaluation exemplar: The critical importance of stakeholder relations in a national, experimental abstinence education evaluation. American Journal of Evaluation, 31(4),  Brandon, P. R., Smith, N. L., & Hwalek, M. (2011). Aspects of successful practice at an established private evaluation firm. American Journal of Evaluation, 32(2),

The Evaluator-Client Relationship How is the evaluator-client relationship related to exemplary work? Brandon, Lawton, & Krohn-Ching - Long term prior relationship with client made study possible. Trenholm, & Devaney - Praised for its design; Exemplary for maintaining effective stakeholder relationships for over 10 years. Hwalek – Client relationships key to effective work.

Necessary Conditions? Are certain conditions necessary for exemplary practice?  Brandon et al., Trenholm & Devaney, and Hwalek - exemplary work was only possible because of certain local conditions; evaluator design and effort were not enough to guarantee exemplary work.  Many conditions have to be right to conduct an exemplary study.  Quality of work is not always under the evaluator’s control, evaluator may not know which factors are most crucial to control.  Sometimes, everything just has to go right for exemplary work to result - the stars have to align (Tolley, 2011).

Recent Work Does what is considered exemplary differ by type of evaluation work? What factors impede or enhance exemplary work? Cousins, J. B., & Chouinard, J. A. (in press). Participatory evaluation up close: An integration of research-based knowledge. Information Age Press. Evaluative review of 106 studies of Participatory Evaluation.

Contextual Factors and Enabling Conditions for Participatory Evaluation Evaluator background and role – culture, expertise, experience, skills. Community context – community characteristics, politics, interpersonal. Institutional influences – support, resources, culture, need. Program influences – complexity, breadth, history, objectives (Cousins & Chouinard, in press, Fig. 2-3).

Obstacles to Participatory Practice Structural – political, organizational, tensions across levels. Administrative – decision making, resources management. Social – lack of resources, dependence. Interpersonal – poor communication, lack of trust (Cousins & Chouinard, in press, pp ).

Next Steps What conditions and actions are required to promote exemplary evaluation use?  AJE collaborative case study with George Grob.  Evaluation myths that evaluations are not used and that methods make the most difference.  “To influence policy makers, new insights on relevant issues, practical and affordable recommendations, easy to read reports, timelines, and personal connections through gatekeepers are what matter” Grob,  Stay tuned!

References Chelimsky, E. (2010, August 10, 16:39). RE:Eleanor Chelimsky, our August Thought Leader [American Evaluation Association Thought Leaders Forum]. Retrieved from lt.aspx?GroupID=91#bm11 lt.aspx?GroupID=91#bm11 Grob, G. (2011). Policy Watch – Getting Through to Policy Makers. AEA Newsletter, 11(10). Retrieved 11/1/11. Tolley, L. M. When the stars align: The fundamental issue of context and its role in exemplary evaluations, paper presented at the annual meeting of the Eastern Evaluation Research Society, Galloway, NJ, May, 2011.