REAL WORLD RESEARCH THIRD EDITION Chapter 8: Designs for Particular Purposes: Evaluation, Action and Change 1©2011 John Wiley & Sons Ltd.

Some purposes of evaluation: 1 To find out if client needs are met likely questions: What should be the focus of a new programme? Are we reaching the target group? Is what we provide actually what they need? continued… 2©2011 John Wiley & Sons Ltd.

Some purposes of evaluation: - continued 2 To improve the programme likely questions: How can we make the programme better (e.g. in meeting needs; or in its effectiveness; or in its efficiency)? continued… 3©2011 John Wiley & Sons Ltd.

Some purposes of evaluation: - continued 3 To assess the outcomes of a programme likely questions: Is the programme effective (e.g. in reaching planned goals)? What happens to clients as a result of following the programme? Is it worth continuing (or expanding)? continued… 4©2011 John Wiley & Sons Ltd.

Some purposes of evaluation: - continued 4 To assess the outcomes of a programme likely questions: How do the costs of running the programme compare with the benefits it provides? Is it more (or less) efficient than other programmes? continued… 5©2011 John Wiley & Sons Ltd.

Some purposes of evaluation: - continued 5 Assessing relative costs and benefits likely questions: How do the costs of running the programme compare with the benefits it provides? Is it more (or less) efficient than other programmes? (from Robson, 2000, Table 1.1, p. 10) 6©2011 John Wiley & Sons Ltd.

Features of evaluation Any evaluation should meet the following criteria: 1 Utility 2 Feasibility 3 Propriety 4 Technical adequacy 7©2011 John Wiley & Sons Ltd.

Checklist for planning an evaluation 1 Reasons, purposes and motivations 2 Value 3 Interpretation 4 Subject 5 Evaluator(s) 6 Methods 7 Time 8 Permissions and control 9 Use (adapted from Robson et al., 1988, p. 85) 8©2011 John Wiley & Sons Ltd.

Skills needed to carry out evaluations writing a proposal clarifying purposes of the evaluation identifying, organizing and working with an evaluation team choice of design and data collection techniques interviewing questionnaire construction and use continued… 9©2011 John Wiley & Sons Ltd.

Skills needed to carry out evaluations - continued management of complex information systems data analysis report writing, including making of recommendations fostering utilization of findings sensitivity to political concerns 10©2011 John Wiley & Sons Ltd.

Needs assessment checklist 1 Identify key individuals to be involved in a steering group 2 Define the target population for the needs assessment 3 Ensure their needs are the focus of the needs assessment 4 Communicate the aims of the needs assessment to service providers 5 Decide who will carry out the needs assessment 6 Consider whether additional assistance may be needed continued … 11©2011 John Wiley & Sons Ltd.

Needs assessment checklist - continued 7 Estimate the cost and identify the source of funding for the needs assessment 8 Identify the appropriate overall approach to your needs assessment 9 Gather existing sources of information about the needs of your target population 10 Identify existing services in your area that are already available to meet the need continued … 12©2011 John Wiley & Sons Ltd.

Needs assessment checklist - continued 11 Consider the ways in which you will obtain the views of your target population about their needs, and whether ethical approval is needed 12 Consider the ways in which you will obtain the views of service providers about the needs of the target population 13 Ensure that information is analysed and interpreted, and that conclusions are drawn 14 Consider how those who gathered the information can be involved in the analysis, and how the results can be relayed back to all those who contributed to the process continued … 13©2011 John Wiley & Sons Ltd.

Needs assessment checklist - continued 15 Once you have identified the needs of your target population, prioritize them, consider all the options for meeting them, and develop an implementation plan 16 Once agreement is reached on the changes to make, consider how to monitor and evaluate them 17 Consider the most appropriate methodology for the evaluation (adapted and abridged from Griesbach et al., 2004, p. 37 ) 14©2011 John Wiley & Sons Ltd.

Stages of action research 1 Define the inquiry 2 Describe the situation 3 Collect evaluative data and analyse it 4Review the data and look for contradictions continued … 15©2011 John Wiley & Sons Ltd.

Stages of action research - continued 5 Tackle a contradiction by introducing change 6 Monitor the change 7 Analyse evaluative data about the change 8 Review the change and decide what to do next (adapted from Bassey, 1998, pp. 94–5 ) 16©2011 John Wiley & Sons Ltd.

Assumptions for those wishing to initiate change 1Don’t assume that your version of what the change should be is the one that could or should be implemented 2Change involves ambiguity, ambivalence and uncertainty 3Some conflict and disagreement are not only inevitable but fundamental to change. 4People need pressure to change 5Effective change takes time continued … 17©2011 John Wiley & Sons Ltd.

Assumptions for those wishing to initiate change - continued 6Lack of implementation isn’t necessarily because of rejection or resistance 7Don’t expect all, or even most, people or groups to change 8You need a plan based on these assumptions and underpinned by a knowledge of the change process 9Change is a frustrating, discouraging business - if you are not in a position to make the above assumptions don’t expect significant change, as far as implementation is concerned. (adapted and abridged from Fullan, 1982, p. 91) 18©2011 John Wiley & Sons Ltd.