1. Tailoring Evaluations
Karen (B.K.) Adler
Every evaluation must be tailored to a specific set of circumstances. – Take into account the purposes of the evaluation, the conceptual and organizational structure of the program being evaluated and the resources available.

2. What Aspects of the Evaluation Plan Must be Tailored?
The questions that need to be answered
The methods and procedure the evaluation will use to answer those questions
The nature of the evaluator-stakeholder realtionship
Questions that need to be answered – Purpose of the evaluation, feedback? Improvement? Info to potential funders? Impact?
Methods – Interplay between the nature of the evaluation situation and the evaluator’s repertoire of approaches and techniques
Nature of stakeholder relationship – use of findings by the various stakeholders? Requires a collaborative interaction – Need to know the audiences that will receive info. – nature and schedule of written reports – How findings are to be disseminated, etc.

3. What features of the Situation Should be Taken Into Account?
Purposes of the evaluation
Program structure and circumstances
Resources available

4. The Purposes of the Evaluation
Program improvement
Program effects
Decisions about funding
Response to political pressures
Companies generally provide a statement of purpose – but these rarely tell the whole story. Sometimes an evaluation is mandated or just seems like a good idea.
Must determine who wants the evaluation, what they want and why they want it.
An evaluator kind of approaches this the way a journalist digs out a story.

5. Purposes of the Evaluation
Program improvement
Accountability
Knowledge generation

6. Program Improvement – Formative Evaluation
Audiences
Program planners
Administrators
Oversight boards
Funders – interested in program effectiveness
Information Desired
Need for program
Program design
Implementation
Impact
Efficiency
Formative evaluation – purpose to help form and shape the program to perform better
Evaluator usually works closely with program management and other stakeholders in designing, conducting and reporting the evaluation.
Findings are generally timely, concrete, and immediately useful.

7. Accountability – Summative Evaluation
Effective use of resources
Beneficial contribution to society
Audiences
Decision Makers
Funding boards
Governing board
Legislative committee
The purpose of a summative evaluation is to render a summary judgement on the program’s performance. When you’re talking about tax dollars – program managers are expected to use resources effectively and efficiently.
Programs need to show that they are producing the intended benefits.
It is important that the evaluation use info that is highly credible under scientific standards because they may influence significant decisions about the program
Evaluator tends to work independently with stakeholders providing input – findings tend to be communicated in a formal setting.

8. Knowledge Generation Test theory Try new approaches Audiences
Sponsors of research
Interested scholars
Policymakers
Evaluations of this sort are intended to make contributions to social knowledge and be a basis for program innovation
Examples – Academic researcher may initiate an evaluation to test a theory – to see if an innovative science curriculum is working Or
Gov. agency investigate a new approach to a social problem – which if successful could be implemented more widely
Findings are generally reported out in journals research monographs, conference papers and other professional outlets.

9. Hidden Agendas Good public relations
Rqationale for a decision that has already been made
Delay tactic to appease critics
Watch out for political and public relations motivations!
Virtually all evaluations involve some political maneuverings and public relations aspects – but if they are the main purpose of the evaluation – the evaluator is in a very difficult position.

10. Program Structure and Circumstances
Stage of program development
Status quo called into question?
Administrative and Political Context of the Program
When major stakeholders don’t agree…
Guba and Lincoln – interpretative dialog among stakeholders
New programs evaluators are often called on to look at the needs addressed, program design, define target population, etc. Evaluator functions as a planning consultant.
Established programs require a greater effort… stakeholders reluctant to change
Food stamp example
Evaluator is not free to define the program on his/her own. Depends on stakeholders to provide the program goals and objectives, questions to be addressed, etc.
When they don’t agree – can focus on one perspective (sponsor) - or try to incorporate multiple perspectives and try to facilitate a better understanding among multiple perspectives. – but it is important to be very up front about the perspective taken so that there are no misunderstandings.
If the basic differences are profound – the stakeholders may turn on the evaluator – there may not be a way to make everyone happy w

11. Conceptual and Organizational Structure of the Program
Program Theory
Sometimes difficult for stakeholders to articulate a program that is well established and routine
Large complex programs – Difficult!
New math curriculum
Discrete or concrete – Easy!
Serve meals to homeless
If the stakeholders don’t have a clear idea about what the program is supposed to be doing – it will be very difficult to evaluate it The more explicit and cogent the program conceptualization – the easier it is for the evaluator.
Large complex programs present greater practical difficulties for the evaluator – may require a team of evaluators Serve multiple target populations, distributed service sites or facilities
Diffuse in nature that take place over a large period of time
Discrete programs – have immediate, observable effects!

12. Resources Available Personnel Materials, equipment
Facilities to support data collection, analysis and reporting
Source of funding? Different aspects of evaluation funded differently?
Importance of breaking down tasks and constructing timelines for accomplishing them.

13. Other Critical Resources
Support from program management
Access to information and program records
Ability to read and understand the records
Cooperation from program personnel
TIME!
Better to answer a few questions well
than a lot poorly
Pretty typical to underestimate the time required.
Trade offs – Evaluation can have breadth, depth and rigor – but will require proportionate time and resources. Cheap and quick evaluations deal with narrow issue and/or are relatively superficial.

14. The Nature of the Evaluator-Stakeholder Relationship
Policy and decision makers
Program sponsors
Evaluation sponsors
Target participants
Program managers
Program staff
Program competitors
Contextual stakeholders
Evaluation and research community
Evaluators should give a great deal of attention to the nature of their relationship with the stakeholders who may participate in the evaluation or have an interest in the evaluation process or results. Every program has a set of political and social relationships among those with an association or interest in the model – policy makers, competing programs and advocacy groups
All or some may be involved in any given evaluation – but it is important to plan apropriate means for interacting with at least the major stakeholders… The top of the list is the evaluation sponsor – This relationship may be directive or collaborative.

15. Some Good Advice Identify stakeholders Involve stakeholders early
Involve stakeholders continuously
Involve stakeholders actively
Establish a structure for the sharing of information.

16. Types of Relationships
Independent Evaluation
Participatory or Collaborative Evaluation
Empowerment Evaluation
Independent Evaluation- Evaluator takes primary responsibility for developing the evaluation plan
Participatory Evaluation – Organized as a team project and the evaluator is a member of the team
Empowerment Evaluation – Emphasizes the initiative, advocacy and self-determination of the stakeholders – is a participatory and collaborative relationship – directed toward developing the capabilities of the stakeholders.
Participation by the evaluation sponsor and stakeholders may insure that the results more closely address their concerns.

17. Communication and Dissemination of Findings.
Varied formats
Short reports, summaries, presentations
Tailor content to the audience
Use clear language
Graphs, charts, vivid, concrete illustrations
Present contextual information
Include both positive and negative findings

18. Evaluation Questions and Methods
Give structure to the evaluation
Lead to appropriate and thoughtful planning
Basis for essential discussions about how the answers will be used
A program evaluation is essentially information gathering and interpretation to answer questions about a program’s performance and effectiveness.

19. 5 types of Questions Needs assessment Assesment of program theory
Assessment of program process
How’s it working – most frequent type of eval.
Impact assessment
Efficiency assessment
Needs assessment – Social conditions program is intended to benefit
Assesment of program theory – Program conceptualization and design
Assessment of program process – Program operations, implementation and service delivery
Impact assessment – Program outcomes and impact
Efficiency assessment – Costs and Cost effectiveness

20. Conclusion
Every evaluation must e tailored to a specific set of circumstances
Purposes of the evaluation
Questions and methods
Resources available
Evaluator-stakeholder relationship