The Value of Rubrics in Messy Non-Profit Evaluation Contexts Kate McKegg American Evaluation Association Conference Washington DC, 2013

Aim of this session Develop an understanding of how the use of rubrics in the non-profit sector can contribute to: Organizational learning Internal evaluation capacity Improved client outcomes 2

What are evaluative rubrics? An explicit way of defining ‘quality’ and ‘value’ for ourselves and for others: – Identifies and clarifies what aspects are important – Identifies criteria to assess how good is ‘good’ – Allows us to make a judgment, using these criteria, about how good, or excellent (or poor) something is.

Evaluative rubrics - a useful tool Makes the ‘values’ step transparent and explicit A broad-brush way of transparently defining what excellent, good, (etc) quality, value or performance would look like in practice A process and framework for being explicit about: – What criteria are used to make evaluative judgements – The types of performance we focus on

General guide for deciding performance (E.J. Davidson 2010)

Mercalli Intensity Scale (1906) for earthquake impacts

Data + rubric & evaluative sensemaking = reasoned evaluative judgment 8

Rubrics and organizational learning 9

10

Ministry of Culture and Heritage & Minister Contributing Schools Auckland Philharmonic Orchestra Programme Director, mgmt, staff & volunteers Community Supporters and Organisations Sistema Families Is this programme worth continuing to fund? How valuable is the programme to the community, to schools, to the children and their families? What can we learn that we can share with others and apply in other settings? How effectively have we implemented the programme How well are we delivering the programme? Are there things we can improve and do better? How effective is the programme in supporting the aspirations and success of our local children? How effectively is the programme contributing to our community? How well does the programme influence children’s learning, behaviour and engagement in education? Is it worth us continuing to support children’s involvement? How well does the programme support my child’s aspirations and wellbeing?

Programme Dimension Evaluation Criteria Content and design Group-based learning incorporates child development theory and practice Reflects the community and its cultural diversity Standardised principles and use of language and processes Clear progression pathway etc Programme Delivery Systems and approaches Human resources Finances and sustainability Relationships etc Outcomes of Value Children Families Schools Community Children experience a sense of accomplishment and success Children experience an increase in their self-esteem, confidence and motivation Families are proud of their children Families and children expand their horizons and their sense of opportunity Children’s behaviour at home, and with their families is more positive The programme makes a contribution to the development of a wide range of new skills, behaviours and competencies The community has a sense of pride and ownership in the programme Communities are positively transformed

Evaluative Criteria guided data collection 13 Data Matrix for determining data collection requirements

14

Data analysis and synthesis using evaluative criteria 15

Generic rubric used for Sistema 16

Participative data synthesis – example data pack info 17

Reporting on the programme’s performance e.g., design, content and delivery

So what happened as a result of using rubrics in this case? Clear direction about where improvements could be made Thank you Kate for this report! It has been great to work with you on this, and I'm looking forward to the next stage ( from Joe Harrop, Programme Director) Clear evaluative judgments that could be reported in a way that even Senior Government Managers and Ministers could understand! And this lead to further funding for the initiative by the Ministry of Culture and Heritage Our thanks… for a thorough report, which confirms the value of this programme ( from Barbara Glaser, Chief Executive, Auckland Philharmonia Orchestra). 19 “The value we’re getting from Sistema is huge. We would never consider pulling away from the programme.” Minister of Culture & Heritage

Rubrics as a mechanism for building organisational and evaluation capacity & improving outcomes 20

21 Evaluation criteria & rubrics driving capacity – org + evaluation

22 Evaluative questions – driving data collection

23 Rubrics provide the organisation with a performance pathway

24 Changes in student outcomes

25 Organizational and Evaluation Capacity

Rubric driving client outcomes 26

Self Managing Cohesive, Resilient and Nurturing Living Healthy Lifestyles Participating Fully in Society Economically Secure Confidently Participating in Te Ao Maori Blood Pressure Weight Waist measurement HBa1c Cholesterol CV risk Clinically Secure Kia Rite Kia Ora – Diabetes Prevention Programme

28 Ladder of whānau change for the outcome Self Managing

29

30

31

Whānau living the dream… We've come almost to the conclusion of our 12 months and we are starting to live the dream. We will continue because this programme has been so awesome, it's made us draw closer to our whānau (family).

Whānau being independent… Kia Rite Kia Ora is about evaluation, independence, reliability, being honest within yourself. And this is where they actually help us to grow and strengthen within ourselves to get out there and be independent, and that’s what this programme is all about. 33

Whānau being whānau… It's not just about us anymore it's about the new addition to the whānau (family) & our moko (grandchildren). It’s given us even more meaning to make all these changes for all of us as a whanau so we can be around as long as possible. 34

THANK YOU Contact Kate McKegg: