1. Introduction to Evaluation Odette Parry & Sally-Ann Baker o.parry@glyndwr.ac.uko.parry@glyndwr.ac.uk s.a.baker@glyndwr.ac.uks.a.baker@glyndwr.ac.uk

2. Aim and objectives of our presentation  Define evaluation and examine how it differs from research  Briefly introduce different evaluation designs and data collection approaches -Strengths and limitations  Developing evaluation plans  Governance and ethical issues

3. What is evaluation?  Evaluation is “a set of procedures to judge a services merit by providing a systematic assessment of it’s aims, objectives, activities, outputs, outcomes and costs.” (NHS Executive, 1997)  Evaluation is learning about ‘what works’ and lessons for future development Evaluation determines how a service is doing

4. So why evaluate?  Provides evidence of: - Is what we are doing working? - What are the benefits and impacts? - What was successful and what not? And why? - Have/are objectives being achieved?

5. So why evaluate?  Provides evidence for: - Stakeholders - Further programme development - Staff development - Other organisations - Funders

6. What do you want to find out?  What is happening and how often?  How is it happening and why is it happening as it is?

7. Approaches  The ‘What and How Much’ question in evaluation addresses measurable ‘OUTCOMES’ associated with Quantitative approaches (e.g. RCTs and Surveys)  The ‘How and Why’ question addresses PROCESS, associated with Qualitative approaches (e.g. semi-structured interviews, observation and focus groups)

8. Horses for Courses  Evaluation usually requires both Outcome & Process information  Evaluation may Formative &/or Summative  Small scale evaluations, while most often qualitative, do collect some quantifiable data

9. Plan your evaluation Evaluation is not a ‘bolt on’ Evaluation key to informing project development and delivery Plan early Involve funders, stakeholders in the planning process

10. Define your purpose  Clearly define the purpose of the evaluation  What are the main aims and objectives This is a key step in the planning process and will guide how the findings will be used

11. Develop an evaluation plan What is being evaluated Purpose of the evaluation What is known already The questions the evaluation will address Evaluation method to be used Who & Where will the evaluation take place Timescales The resources available

12. Be realistic about what can be achieved Often evaluation is a compromise between the ideal and the achievable - the wants of different groups - constraints of methodology - evaluator skills - resource limitations - time limitations - ethical and governance issues

13. Different ways to collect data  Decide which methods to use in order to get the information you need. You may use one or more of the following: - Existing data - Document analysis - Interviews - Self completed questionnaires - Observation - Focus Group

14. Using existing data  Use data routinely collected to examine process and outcomes  Can save time but need to ensure that data collected, in a form that can be analysed and consistent with evaluation plan  Evaluation of All Wales Dietetics Scheme used Minimum data sets. Developed in conjunction with WAG and project dieticians to ensure consistency with evaluation aims  Collected data on project activity e.g. courses run, start and completion dates, description of activities, participant details, course outcomes, partnerships built etc

15. Document Analysis  Policy documents, minutes, operational policies - Track project development, aims of project  Whilst can provide rich information  May only provide part of picture  May be open to subjective interpretation

16. Observation  Observing participants in activities  Participant vs non participant  Observer effects  Not suitable for some settings  Can be difficult to collect & record data

17. Interviews  Enables in depth exploration of how people think & feel about certain topics, effectiveness of your intervention etc  Rich data, allows in-depth understanding  Can explore more sensitive area  Can tailor to the individual  No group influence  Resource intensive

18. Focus groups  Investigation of how groups perceive topics, view the effectiveness of your intervention  Clarify issues identified in surveys  Provide solutions to problems  Less resource intensive and 1 to 1 interviews  Group management issues  Group process issues

19. Questionnaires  Identification of patterns, trends  Investigate needs, expectations, perspectives, preferences, satisfaction, knowledge  Large samples, relatively lower cost  Ease of analysis  Low response rates  Respondent bias  Language and literacy issues

20. Analysing your data  Avoid bias, involve more than one person in the task  Address your key questions  Combine data types & findings from different sources  Compare views of different groups  Don’t anticipate results, look for unexpected findings

21. Evidence of impact Focus on whether:  the purpose has been achieved  the needs of those who take part have been met  there are unintended outcomes arising from the intervention  the intervention has resulted in changes in behaviour  there are barriers to and facilitators of successful implementation

22. Using your findings  Inform strategy, policy development  Inform budgets  Inform funding proposals  Inform improvement plans & and make changes  Identification of training needs  Identification of areas for future research/evaluation

23. Presenting evaluation findings  Present balanced view, don’t just report the positive  Consider the needs of your audience,  Different versions of reports or types of presentation might be needed

24. Reflect on the evaluation process  Were aims and objectives met? If not why  Were methods employed appropriate?  Tools, recruitment methods, analysis etc  Did you reach target groups?  Were resources sufficient?  What changes resulted ?  Future impacts?

25. Governance Issues  Be careful about crossing over from evaluation into the realms of research – carries additional connotations for ethics approval etc  Take advice  Take an ethical/professional approach to the evaluation, consider the participant and yourself

26. Ethics and the rights of Participants  Requirements of Data Protection Act (1998)  Some evaluations may require ethical approval Issues to consider - Informed Consent - Deception - Debrief - Withdrawal - Confidentiality

27. Data Protection Act 1998 – Your responsibility  Only hold genuinely required information  Certain types of sensitive information carry added restrictions and may only be used with permission  Information should only be collected for the purposes for which it was initially required  Data must only be processed in accordance with the legislation  People must not be harmed by how their information is used  People have a right to view information being held about them  Information must be accurate, kept up to date, kept secure and deleted when obsolete

28. In summary  Evaluation an essential component  Need to plan  Variety of approaches  Methods used dependent on questions to be asked  Consideration of Governance issues

29. Resources  Data Protection Act see the information commissioners pages http://www.ico.gov.uk/what_we_cover/data_protection. aspx http://www.ico.gov.uk/what_we_cover/data_protection. aspx  National Research Ethics Service if your evaluation takes place on NHS premises or with Staff, patients ethical approval might be needed check with your local R&D office and LREC administrator http://www.nres.npsa.nhs.uk/applications/apply/ http://www.nres.npsa.nhs.uk/applications/apply/  UK Evaluation Society has a useful resource page http:// www.evaluation.org.uk/resources/online-resources.aspx http:// www.evaluation.org.uk/resources/online-resources.aspx