1. DAVIDSON CONSULTING LIMITED
Collaboratively defining and evaluating ‘quality’ and ‘value’: A participatory evaluation and capacity building process for Learning Support & Resource Centres
World-class evaluation & organisational consulting
E. Jane Davidson, Yvonne Hope & Judith Nel
Davidson Consulting Ltd Group Special Education Parkside Special School
aotearoa new zealand evaluation association (anzea) conference,
Masterton, New Zealand, July 9, 2007
New Zealand Association of Special School Principals SS

2. Overview Introduction to Learning Support and Resource Centres (LSRC)
The evaluation approach – intent, purpose, philosophy, practice
Evaluation nuts and bolts – framework, tools, methods and capacity building
The shared journey: A user’s perspective
Run video at this point

3. Background to LSRCs Pre SE 2000 Transition arrangements
How to continue and enhance an itinerant service?
Proposal to Minister
Collaborative development

4. Evaluation approach Original plan: Contract to external evaluator
Rethink based on the needs and philosophy:
Collaborative approach to LSRCs => collaborative approach to evaluation
Need to document and share lessons learned
Need robust evidence of effectiveness/value
Must be practical, user-friendly, quick to use and have high ‘face validity’
Need to build ‘learning capacity’ within the sector
Solution: Facilitated, high involvement process that combined evaluation expertise, special education expertise and front-line experience

5. Evaluation nuts and bolts
Development of a logic model to show how LSRCs should work to achieve key outcomes
Development of ‘rubrics’ that define what evidence would constitute ‘excellent’, ‘good’, ‘just adequate’, etc. on all elements
Identification of five key outcomes to track across all LSRCs
Defining ‘value’ using evaluative rubrics
Getting value from the evaluation
Generating reports from the data

6. Logic Model

7. Identification of five key outcomes
GSE, in consultation with the evaluation development group, has identified five key outcomes on which data need to be collected annually using the developed rubrics:
3.2 The child receives an effective, tailored development programme coordinated by a skilled specialist teacher who documents and uses learner outcomes to inform practice
3.4 School culture is inclusive, welcoming and positive
4.1 Student learning
4.3 Student presence, participation and engagement
4.4 Student is actively involved in their community
LSRCs report on the five key outcomes (3.2, 3.4, 4.1, 4.3 and 4.4) as a bare minimum, using the five rubrics to gather data for at least 5 to 10 students (or all students, if fewer than 5).

8. Rubric: Presence, Participation & Engagement
Rating
Description
Tick ()
Comments on Evidence*
Excellent
Student is positively engaged in interpersonal relationships, social activities and shared learning experiences and is increasingly self-determining. Student is a valued member of the school community and may have some friendships. 
Prior to the LSRC initiative the student was attending two half-days per week; currently (after two years) the student is attending five full days per week. The mother was able to start a job thanks to this improvement.
* Key pieces of evidence used to arrive at the rating. Include a short explanation of any variance from the description in the rating level chosen.
Very Good
Student is frequently included in shared learning experiences and social activities. There is evidence of emerging positive interpersonal relationships.
Good
Student is participating and encouraged in appropriate classroom and playground opportunities and interactions. 
Just
Adequate
Student is attending school and participating in some appropriate classroom and playground activities and interactions.
Inadequate
Student is restricted from attending or participating in school activities or unacceptably high levels of unexcused absences or levels of participation are far lower than the child would be capable of.

9. Getting value from the evaluation
Initial trials of the logic model and the rubrics revealed several ways in which they can be used as valuable tools for LSRCs. These include:
As a promotional and discussion document to explain and clarify what the initiative is seeking to achieve
To focus and motivate a team with a clear vision of what excellence looks like, and to provide exemplars for improved performance and celebrate success
To track individual student progress and identify ways to improve student outcomes
As evidence to assure parents, Boards of Trustees, schools and the Ministry that the initiative is achieving results
As a model, methodology and set of tools that can be adapted for use in other projects

10. Generating reports from the data
Data gathered on a selection of rubrics were used to generate:
A longitudinal milestone report showing progress on key outcomes over a 6 or 12 month period
A half-yearly report on individual student progress, to go to parents and schools

11. Longitudinal Milestone Report
Report includes:
Summary of outcomes (include graphs showing professionals’ and family/whanau perspectives)
Analysis/explanations
Highlights
Issues arising

12. Individual Student Report
Report includes:
Student outcomes (graphs used to give summary of progress from both professionals’ and family/whanau perspectives)
Details on student learning; presence, participation & engagement; community involvement
Summary of performance on IEP goals

13. Key outputs, outcomes and learnings
Simple, user-friendly framework and tools
Data are being used to improve the quality of implementation and outcomes for students
Data are persuasive and compelling to key stakeholders
Evaluation development team upskilled to pass on the know-how to others
Data and analysis replace and improve on existing milestone reports (not ‘additional’)
Next stage (2008) is to share the framework, tools and know-how with other LSRCs

14. The shared journey: A user’s perspective