1. Evaluation Workshop Self evaluation – some workable ideas and approaches

2. Why self-evaluation? Reflexive practice – knowing what has been successful or unsuccessful and why –Evidence for beneficiaries –Evidence for commissioners or partners Practitioner-led research - encouraged by the Children’s Workforce Development Council Strengthening the wider evaluation arrangements for the 0-7 Pilot Partnership

3. Self evaluation: the stages involved

4. Getting started The Action Learning Sets: –Aims –Objectives –Milestones –Outcomes Leap of faith from testing the activities to proving that certain outcomes have been achieved Theory of Change model breaks this down…

5. 1.Start here… The problem or issue to be addressed 7. Impacts The ‘Big’ / wider effects that are brought about 2. Theory of Change The idea or approach for tackling the problem 3. Inputs Resources; time or money to be used 4. Activities What will actually be done, and how 5. Outputs The end result of the activities 6. Outcomes What will be different for the people or service that you are trying to improve Theory of change…

6. What to measure? Hard outcome: a clearly defined (quantifiable) change ‘The family learning course improved my literacy and numeracy skills’ Soft outcome: a harder to measure (qualitative) change ‘The family learning course made me more confident in speaking about my experiences with other parents’ Indicator: a sign that change has occurred over time Numbers of new enrolments Percentage (%) of parents completing OCN Level 1

7. Showing that change has happened Baseline Time Chosen Indicator e.g. % of parents reading to their child every week End of project Start of project

8. Gathering the information

9. 1. Secondary evidence Existing sources of information to show what has been done / achieved by the project Think - what data is already available? Core service data * School visitor records Child or parent-held records * Staff observations / CPD CAF records * Partner evidence What else do you need, on top of this? Adding to your records Project-specific monitoring

10. Practical suggestions Have something in place from the start –Much harder to collect evidence after the event –Need a ‘baseline’ to measure what has changed Who will be responsible for gathering the information and how often? –Realistic approach –Match it to the available time and resources Don’t change the format half way through…

11. 2. Primary evidence New or additional information to show what has been achieved, e.g. insights, personal accounts, experiences and case studies Who needs to be consulted? –Stakeholders –Beneficiaries of your project Staff, parents, children Who will gather the information and how? –Self-reporting / gathered by someone else

12. Examples… Self-reporting Feedback sheets Comments boxes Research diaries Pictorial or video evidence SMS / Text or email feedback Blogs Gathered directly Observations Structured interviews or focus groups Action research / practitioner research Piggy-backing: –Involve existing parent or community panels –Work into annual resident’s surveys

13. Some key points Structure is needed - what are you asking people to feed back? –Structured doesn’t have to mean formal –Satisfaction is different to outcomes Impartiality and avoiding bias –Are you too close to ask the questions? –Who else could do this instead? Volunteers / parent advocates / partner staff CRB checks and Health and Safety Confidentiality and informed consent –Do participants know how you intend to use the information?

14. Questionnaires – some tips Make them confidential Language used – support for completion Appearance / length Style of questions – avoid bias Was the session: –A) Excellent –B) Very good –C) Good x

15. Questionnaires… Overall, how would you rate your personal confidence, now? Tick the box that describes you. a) I am confident in myself (4) b) I struggle a bit with my confidence (3) c) I need a lot of help with my confidence (2) d) I have very low confidence (1) How far would you agree with the following statements? Strongly agree (4) Agree (3) Disagree (2) Strongly disagree (1) Don't know a. "I feel nervous about using childcare services" b. "I am confident about my child's development" c. "I wish I knew more about services for families in my area"

16. Analysing and reporting on evidence

17. Analysing and reporting 1) Qualitative evidence Case studies Individual or service level Telling a story: –what was the issue? –how was it tackled? –what was successful? –what is different now? Using quotes Powerful form of evidence Confidentiality 2) Quantitative evidence Reporting survey findings Less can be more Non-numeric is often best:: –‘approaching half’ –‘nearly a third’ –‘a sizeable majority’ Charts and tables Presenting a clear message Outcomes for different groups

18. Support from ECOTEC…

19. Support from ECOTEC Evaluation Toolkit –Key principles of doing evaluation –A source of ideas and examples –Benchmarking what you are doing – good practice examples One-to-one advice –Commenting on your individual evaluation plans –Q&A and troubleshooting via the support email / telephone Sharing findings from the independent evaluation –Interim and final reports –Case studies

20. Thank you for listening