The Value of Rubrics in Messy Non-Profit Evaluation Contexts Kate McKegg American Evaluation Association Conference Washington DC, 2013.

Slides:



Advertisements
Similar presentations
Engaging with Commissioning Heidi Bellamy - Director, Culture First Fiona Marriott - Adult Services Manager, Luton Culture.
Advertisements

Implementing NICE guidance
LIFE, LEARNING AND ACHIEVEMENT
Te Tukunga Iho o Te Pu o Te Wheke National Whānau Ora Hui Holiday Inn, August 2011 Te Pu o Te Wheke.
Personal, Social, Health and Economic education How PSHE education contributes to meeting the requirements of the Secondary National Curriculum.
Healthy Schools, Healthy Children?
Evaluating and Measuring Impact Presented by – Date –
David Taylor Formerly Director of Inspection, Ofsted
The Index for Inclusion. Why have an Index Forum? Purpose To offer regular opportunities to discuss school improvement with other neighbouring schools,
Catulpa Community Support Services.  Use of an electronic data entry program to record demographic data and case notes to reflect service delivery 
Kate Moorse – Programme Leader, KS3 Curriculum October 2005 Key stage 3 review.
Transforming health and social care in East Sussex East Sussex Better Together.
Influencing and challenging mainstream practice- A Scottish perspective. Monica Mc Geever HM Inspector Education Scotland.
Effective support: working with others Effective support: working with others A Twilight Training Session by Gareth D Morewood, Director of Curriculum.
Some problems are simply too complex to solve with any single approach. Why collaborate? Isolated interventions and single programmes will never be able.
WELCOME TO THE GREAT EXPLORATION! CRITICAL REFLECTION ON SELF AS LEADER AND LEADERSHIP PRACTICE.
Partnership between the NYCI, the Health Service Executive and the Department of Children and Youth Affairs National Youth Health Programme Kevin O’ Hagan.
Gilgandra High School School Development Day, Term
A big picture of the curriculum Adapted with thanks to colleagues at the Council for Curriculum, Examinations and Assessment (CCEA) Working draft: With.
Taranaki Whānau Ora Hui Owae Marae, Waitara Tuesday 1 st June 2010.
Partnership Working in Linburn Academy. Young People With Complex Learning Needs.
The Vision Implementation Project
1. Why staff wellbeing must be part of Healthy FE Sam Mellor, Department of Health.
National Standards and gifted and talented: are they compatible? Presented 10 Nov – 5pm Webinar 4 of a series of 4 Adrienne Carlisle Evaluation.
Curriculum for Excellence Aberdeenshire November 2008.
1 School Inspection Update Key Changes since January 2014 Updates continued 17 June 2014 Name Farzana Aldridge – Strategic Director & Caroline Lansdown.
1 CQC – the next phase Alan Rosenbach Special Policy Lead.
Family learning has the potential to reach new adult learners who would not be engaged in any other way and whose ambitions are varied. Angela West Family.
Annual Public Meeting 1 September, ASB Community Trust  Established in 1988 as a result of the sale of the Auckland Savings Bank  15 Trustees.
Implementing the Scottish Government’s Strategic Guidance for Community Learning & Development The Voluntary Sector Role Third Sector Interface conference,
The Teacake Club Building a learning community of front line welfare to work staff Richard Scothorne, Rocket Science Faye Thomas, Cardinalis.
Transforming lives through learning Profiling Education Scotland.
Unpacking the ‘Parenting in Wales: Guidance on engagement and support’ 10 September 2014 Workshop 3 Gail Bennett, Flintshire Parenting Strategy Coordinator.
Better Business Cases “Investing for change” Overview
Manchester Museum Kate Glynn Volunteer Coordinator The Manchester Museum / Whitworth Art Gallery.
With libraries, registration & archives An introduction to our services and the way we do things Cath Anley – April 2012 connect.
Organisational Journey Supporting self-management
DAVID WOLFF, DIRECTOR Healthy Partnerships. Cupp Aims “to become recognised as a leading UK university for the quality and range of its work in economic.
Opening Activity  Welcome!  Sit at a table that represents an area on which you would like to talk with others about  As you sit at your table, waiting.
A Focus on Health and Wellbeing Wendy Halliday Learning and Teaching Scotland.
Partnership, Voice & Excellence Te Ao Māramatanga (College of Mental Health Nurses) Māori Caucus Te Rau Matatini- National Māori Health Workforce Development.
Including Parents Conference The shaping of Children’s Services in Nottinghamshire Anthony May Corporate Director for Children, Families and Cultural Services.
Effective Networking for Social Learning The Experience of Grupo Chorlaví Julio A. Berdegué Presented at the Annual Meeting 2004 of Euforic, June 8-9,
Public Health Dorset Presents Rhonda Halling, May 2015.
Integral Health Solutions We make healthcare systems work in harmony.
Kathy Corbiere Service Delivery and Performance Commission
Engaging with Commissioning Heidi Bellamy Policy Development & Business Manager.
Growing Health: The health and wellbeing benefits of community food growing How the health service can use food growing to deliver.
Developing Monitoring & Evaluation Frameworks: Process or Product? Anne Markiewicz.
Creating an evaluation framework to optimise practice returns: Process & progress within a large community sector organisation Australasian Evaluation.
What is a Health Promoting School? ‘Health Promoting Schools are schools which display, in everything they say and do, support and commitment to enhancing.
Introducing Improving Quality Together. Purpose Improving Quality Together aims to support a change in mindset in NHS Wales, where each individual demonstrates.
Building Capacity and Culture within a Research & Evaluation Team anzea July 2007 Research & Evaluation Team Research Division.
The Workforce, Education Commissioning and Education and Learning Strategy Enabling world class healthcare services within the North West.
To Learn & Develop Christine Johnson Lead Nurse Safeguarding (named nurse) - STFT Health Visitors Roles and Responsibilities in Domestic Abuse.
STRONG FAMILY AND COMMUNITY ENGAGEMENT IS CENTRAL TO EFFECTIVE SCHOOL REFORM Jan Patterson and Ann Bliss Smarter Schools National Partnerships Key Reform.
People lives communities Preparing for Adulthood Getting a good life Contribution through volunteering Julie Pointer Preparing for Adulthood March 2016.
Ko te reo te tuariki. Language is my identity. Ko te reo tōku ahurei. Language is my uniqueness. Ko te reo te ora. Language is my life.
Middle Managers Workshop 2: Measuring Progress. An opportunity for middle managers… Two linked workshops exploring what it means to implement the Act.
Welcome to Southern Health Southern Health exists to improve the health, wellbeing and independence of the people we serve.
Here’s something you’ll already know Curriculum for Excellence is designed to support young people to develop the four capacities.
Improve Outcomes in Literacy
New Zealand Health Strategy One Team: Where to start, what to do?
Learning for Living Bruce Lynch Project Manager Foundation Learning
The Scottish Education System
Competency Based Learning and Development
Talent Management and Leadership Development for the Health Workforce
Primary PE and Sport Premium
Understanding your Impact on Well-Being of Future Generations (Wales) Act 2015 Contributing to and Reporting on the 7 Wellbeing Goals, Local Wellbeing.
Quality Framework Overview
Presentation transcript:

The Value of Rubrics in Messy Non-Profit Evaluation Contexts Kate McKegg American Evaluation Association Conference Washington DC, 2013

Aim of this session Develop an understanding of how the use of rubrics in the non-profit sector can contribute to: Organizational learning Internal evaluation capacity Improved client outcomes 2

What are evaluative rubrics? An explicit way of defining ‘quality’ and ‘value’ for ourselves and for others: – Identifies and clarifies what aspects are important – Identifies criteria to assess how good is ‘good’ – Allows us to make a judgment, using these criteria, about how good, or excellent (or poor) something is.

Evaluative rubrics - a useful tool Makes the ‘values’ step transparent and explicit A broad-brush way of transparently defining what excellent, good, (etc) quality, value or performance would look like in practice A process and framework for being explicit about: – What criteria are used to make evaluative judgements – The types of performance we focus on

General guide for deciding performance (E.J. Davidson 2010)

Mercalli Intensity Scale (1906) for earthquake impacts

Data + rubric & evaluative sensemaking = reasoned evaluative judgment 8

Rubrics and organizational learning 9

10

Ministry of Culture and Heritage & Minister Contributing Schools Auckland Philharmonic Orchestra Programme Director, mgmt, staff & volunteers Community Supporters and Organisations Sistema Families Is this programme worth continuing to fund? How valuable is the programme to the community, to schools, to the children and their families? What can we learn that we can share with others and apply in other settings? How effectively have we implemented the programme How well are we delivering the programme? Are there things we can improve and do better? How effective is the programme in supporting the aspirations and success of our local children? How effectively is the programme contributing to our community? How well does the programme influence children’s learning, behaviour and engagement in education? Is it worth us continuing to support children’s involvement? How well does the programme support my child’s aspirations and wellbeing?

Programme Dimension Evaluation Criteria Content and design Group-based learning incorporates child development theory and practice Reflects the community and its cultural diversity Standardised principles and use of language and processes Clear progression pathway etc Programme Delivery Systems and approaches Human resources Finances and sustainability Relationships etc Outcomes of Value Children Families Schools Community Children experience a sense of accomplishment and success Children experience an increase in their self-esteem, confidence and motivation Families are proud of their children Families and children expand their horizons and their sense of opportunity Children’s behaviour at home, and with their families is more positive The programme makes a contribution to the development of a wide range of new skills, behaviours and competencies The community has a sense of pride and ownership in the programme Communities are positively transformed

Evaluative Criteria guided data collection 13 Data Matrix for determining data collection requirements

14

Data analysis and synthesis using evaluative criteria 15

Generic rubric used for Sistema 16

Participative data synthesis – example data pack info 17

Reporting on the programme’s performance e.g., design, content and delivery

So what happened as a result of using rubrics in this case? Clear direction about where improvements could be made Thank you Kate for this report! It has been great to work with you on this, and I'm looking forward to the next stage ( from Joe Harrop, Programme Director) Clear evaluative judgments that could be reported in a way that even Senior Government Managers and Ministers could understand! And this lead to further funding for the initiative by the Ministry of Culture and Heritage Our thanks… for a thorough report, which confirms the value of this programme ( from Barbara Glaser, Chief Executive, Auckland Philharmonia Orchestra). 19 “The value we’re getting from Sistema is huge. We would never consider pulling away from the programme.” Minister of Culture & Heritage

Rubrics as a mechanism for building organisational and evaluation capacity & improving outcomes 20

21 Evaluation criteria & rubrics driving capacity – org + evaluation

22 Evaluative questions – driving data collection

23 Rubrics provide the organisation with a performance pathway

24 Changes in student outcomes

25 Organizational and Evaluation Capacity

Rubric driving client outcomes 26

Self Managing Cohesive, Resilient and Nurturing Living Healthy Lifestyles Participating Fully in Society Economically Secure Confidently Participating in Te Ao Maori Blood Pressure Weight Waist measurement HBa1c Cholesterol CV risk Clinically Secure Kia Rite Kia Ora – Diabetes Prevention Programme

28 Ladder of whānau change for the outcome Self Managing

29

30

31

Whānau living the dream… We've come almost to the conclusion of our 12 months and we are starting to live the dream. We will continue because this programme has been so awesome, it's made us draw closer to our whānau (family).

Whānau being independent… Kia Rite Kia Ora is about evaluation, independence, reliability, being honest within yourself. And this is where they actually help us to grow and strengthen within ourselves to get out there and be independent, and that’s what this programme is all about. 33

Whānau being whānau… It's not just about us anymore it's about the new addition to the whānau (family) & our moko (grandchildren). It’s given us even more meaning to make all these changes for all of us as a whanau so we can be around as long as possible. 34

THANK YOU Contact Kate McKegg: