intandem evaluation

intandem evaluation Presentation contents Quick exercise The intandem approach to data collection - evaluation The intandem approach to monitoring Recording the data External evaluation

intandem evaluation – quick exercise In pairs (or in threes) identify five different groups who would be interested in knowing the impact the service is having – and explain why they would be interested.

intandem evaluation – quick exercise Children How much progress are they making? Where might support help them? Volunteers How much difference are they making for their mentee? Are they addressing the child’s needs? Co-ordinators Are the volunteers making a difference? Are there some mentors or mentees who need more support? Do we need to review how we train/support our mentors? Service managers Is the mentoring service making a difference? Are there ways of improving the service? Charity Board Are we delivering what we set out to deliver? Is the mentoring service effective? Inspiring Scotland Is the mentoring service effective right across the portfolio? Where there are differences, do we understand them? Scottish Government Is the funding achieving what was expected? Does the impact measured justify funding for a wider group?

intandem evaluation – five core outcomes Children have increased confidence Children have improved self-esteem Children have improved social relationships Children are more engaged with the local community Children experience reduced isolation 1 2 3 4

intandem evaluation – confidence and self-esteem Confidence is a belief in being able to do things – and confidence can vary from one area of your life to another. I am confident I can cycle 40 miles in a day, but I am not confident to speak in front of people without notes. Self-esteem is how you see yourself in the world, what value you place on yourself. Self- esteem tends to be more consistent across different areas.

intandem evaluation – confidence indicators I know the things I am good at and enjoy doing them I enjoy trying to do new things I am good at asking for help I can cope well in new and unexpected situations I am comfortable meeting new people All scored on the basis of Strongly Agree (4), Agree (3), Disagree (2) and Strongly Disagree (1), and on the assumption that we are talking about how the child feels most of the time.

intandem evaluation – self-esteem (Rosenberg) I feel that I am as good as other people I think there are lots of good things about me Sometimes, I feel like I am a failure

intandem evaluation – self-esteem (Rosenberg) I am able to do things as well as most other people I like being me I feel good about being me Sometimes I think I am no good at all I feel I do not have much to be proud of Sometimes I feel useless I wish I liked myself more

intandem evaluation – self-esteem (Rosenberg) I feel that I am as good as other people I think there are lots of good things about me I am able to do things as well as most other people I like being me I feel good about being me Sometimes I feel like a failure Sometimes I think I am no good at all I feel I do not have much to be proud of Sometimes I feel useless I wish I liked myself more What do you think? Reliability, validity and appropriateness or fewer questions and softer statements.

intandem evaluation – social relationships and isolation People who care about me People I can talk to about anything People I really like being with People I talk to and play games with

intandem evaluation – community engagement Four levels of engagement: 0. Zero awareness Awareness Access Participation Ownership / leadership Across five areas of community: Sports facilities In school (out of hours) activities Community centres/youth club/libraries Medical and dental services Local facilities – shops, McDonald’s, cinema

intandem evaluation – community engagement Sports Out of school Entertainment Health Shops 1 2 3 4

intandem evaluation – when and who? Baseline data: as soon as practical after the referral has been accepted by you: just knowing that they will be getting a mentor could have an impact. So, at first home visit, by the co-ordinator. Difference measurement: every six months – from the date of the baseline data. So, within the mentoring relationship, by the volunteer mentor.

intandem evaluation – recording the data Use your own systems initially Small numbers, so no great administrative issue Always keep the source paper work until information transferred We are reviewing options for a web-based approach to recording this data Keep a note of those children whose baseline data is captured before the web- based approach is available

intandem evaluation – other data Recording progress against individual objectives Significant SHANARRI indicators e.g. safe Monitoring data for the children Monitoring data about the mentors What do you think? Should we track achievement of individual objectives? If so, how should we do this?

intandem evaluation – monitoring data: mentees Age Gender Time as LAC@H before matching Local authority Post code Living in: rural/village, small town, large town/city Number of parents at home; number and age of siblings at home Outcome: matched or not matched (if not matched, reason for this) Match history: start date, end date, name of mentor, typical meeting time, activities undertaken Education: attendance records and instances of exclusion; educational attainment during period of match

intandem evaluation – monitoring data: mentors Age Gender Employment status: full-time, part-time, student, unemployed, retired Local authority Post code Initial commitment (in months) Additional training attended Outcome: matched or not matched Match history: start date, end date, age/gender of mentee, travel time to mentee, transport used to meet mentee

intandem evaluation – external evaluation In April we will be discussing with potential evaluators what is required for a successful evaluation of intandem. What we already know is we need: Fidelity (all mentors volunteers, same frequency of meetings) Consistency in data collection What do you think? How can we ensure that the data is collected consistently if we use the volunteers to be involved in collecting it? And, what do you need from us?

intandem evaluation Questions and Comments…………….. ……………then lunch.

intandem evaluation – Rosenberg self-esteem

intandem evaluation – summary

intandem evaluation – external evaluation