1. Demonstrating outcomes and impact in the context of professional learning and development

2. Aims
To identify and explore some key questions regarding evaluation, impact and outcomes
To introduce and exemplify a 'level model' of evaluation of professional development outcomes

3. Impact evaluation should focus on what participants learn, how they use what they have learned and the effect on the learning of children and young people (Training and Development Agency for Schools 2007, p. 2).

4. Kirkpatrick (1998),identified four levels of outcome for training interventions: (i) participants’ reactions; (ii) participants’ learning; (iii) changes in participants’ behaviour; and (iv) desired results.

5. Measuring impact and outcomes: key questions
Which aspect(s) of impact are we measuring?
Are these the right ones?
For what purpose are we measuring them?
How do we go about measuring them?
Are our measurements valid?
What do we learn from measuring them?

6. A levels model of evaluation of professional development outcomes
We have used the term ‘level models’ to describe a family of evaluation approaches that share the characteristic of tracing the effects of training and development interventions through a series of ‘levels’, each of which more closely approaches the ‘ultimate’ intentions or outcomes of the intervention.
Coldwell, Mike and Simkins, Tim (2010) page 2

7. Level model Coldwell and Simkins 2010
Interventions
CPD input
Intermediate Outcomes 1 Participant reactions
Intermediate Outcomes 2 Learning & personal development
Intermediate Outcomes 3 Behaviour
Final Outcomes 3 Capacity individual/team capability Education setting systems/practices/culture
Final Outcomes 2 Career development
Final Outcomes 1
Short/longer term CYP outcomes
Antecedents Participants' expectations and motivation
Moderating factors Setting/organisational culture Participants' role/ organisational structure Capacity/resourcing to support change Intervention support - coaching, supervision Time to reflect/plan/act

8. Interventions: the CPD activities themselves
Interventions: the CPD activities themselves. Antecedents: those factors associated with individual participants that affect their ability to benefit from the opportunities offered to them. Intermediate outcomes: those outcomes of the CPD activities that are steps towards (pre-conditions for) the achievement of the final outcomes, Final outcomes: the intended effects of the CPD activities, on organisations, teachers and children and young people.

9. Moderating factors: variables in the school and wider environment that influence whether, and how, the interventions lead, via the achievement of intermediate outcomes to producing final outcomes. These factors help to explain why apparently similar activities have different consequences for different individuals, teams and schools. Understanding these factors helps in preparing/ supporting individuals and settings to achieve intermediate and final outcomes

10. Narrowing the Gap: Improving Learning for Vulnerable Children and Young People using Lesson Study
TSA supported and funded project - context school improvement, raising attainment for vulnerable groups of children
Representative schools identified in clusters by TSA lead group (primary secondary special)

11. Interventions
Session 1 Introduction to lesson study
Identify a pupil who puzzles you and is not making adequate progress
Identify lesson study team in school
Session 2 Planning the lesson study
Supported planning addressing main issues of implementation
Inter session offer of additional support for planning
Session 3 Outcomes presentations
Sharing practice agreeing outcomes analysis and moderating factors
Antecedents
Individual school representation - a team must include SLT and an inclusion specialist e.g. SENCO
Participants' expectations and motivation - briefing paper sent out outlining lesson study and the purpose of the project
Intermediate Outcomes 1 Participant reactions
Identifying learners
Commitment to undertake lesson study
Active initial planning as a team

12. Intermediate Outcomes 1 Participant reactions
Identifying learners
Commitment to undertake lesson study
Active initial planning as a team
Intermediate Outcomes 2 Learning & personal development
Understanding the key features of the lesson study model
Developing the practice of working collaboratively with colleagues
Developing the practice of gaining learner's perspectives on their classroom experiences
Intermediate Outcomes 3 Behaviour
Teachers /TAs apply knowledge of individual learners and insight into learner perceptions of learning
Focussed collaboration between teaching assistants, teachers, SENCO and specialists
Teachers /TAs shift emphasis from techniques re task completion/ curriculum delivery to barriers to /enablers of learning
Active learner engagement in goal-setting and understanding their own learning and influencing their learning context
Moderating factors Setting/organisational culture - is this a priority for us?
Capacity/resourcing - this is resource intensive, do we have the resource and are we willing to commit it to this work?
Time to reflect/plan/act
Moderating factors Setting/organisational culture - are we focused on understanding learners learning or implementing an intervention
Participants' role - SLT support? Parental support for the lesson study
Capacity/resourcing - staffing and attendance issues
Time to reflect/plan/act - has the team applied the features of lesson study effectively? Did they need/ access more support to do so?

13. Final Outcomes 3 Capacity individual/team capability Education setting systems/practices/culture
Enhances teachers’ /TAs’ awareness of the importance of knowledge of individual learners and insight into learner perceptions of learning
Develops skills in gaining learners' perceptions regarding their learning
Challenges teachers’ /TAs’ perceptions of learners
Models structured collaboration between teaching assistants, teachers, SENCO and external specialists
Final Outcomes 2 Career development Implement and articulate:
leadership of collaborative practice
evidence informed responses to differentiation, personalisation, intervention and adjustment
Final Outcomes 1 Short/longer term outcomes for learners
Identifies barriers to learning that impact on a range of learners
Informs design of interventions, adjustments and associated support
Enables progress towards short and medium-term targets
Moderating factors Culture- preconceptions regarding learners' role, autonomy and insights
Capacity/resourcing to enable collaboration
Time to reflect/plan/act re interventions/ adjustments etc 
Intermediate Outcomes 3 Behaviour
Teachers /TAs apply knowledge of individual learners and insight into learner perceptions of learning
Focussed collaboration between teaching assistants, teachers, SENCO and specialists
Teachers /TAs shift emphasis from techniques re task completion/ curriculum delivery to barriers to /enablers of learning
Active learner engagement in goal- setting and understanding their own learning and influencing their learning context

14. References
Coldwell, Mike and Simkins, Tim (2010) 'Level models of continuing professional development evaluation: a grounded review and critique', Professional Development in Education pp1-15
Kirkpatrick, D., (1998) Evaluating training programmes: the four levels. 2nd ed. San Francisco, CA: Berrett-Koehler
Sachs, Judyth (2007) Learning to improve or improving to learn: the dilemma of teacher continuing professional development, Proceedings of the International Congress Effectiveness and Improvement.
Training and Development Agency for Schools, Impact evaluation of CPD. London: TDA.

15. Professional development/learning Sachs 2007
Retooling
Remodelling
Revitalising
Re-imagining
Driver
Accountability and control by government
Compliance with government change agenda
Professional renewal
Professional reinvention
Purpose
Upgrading of skills
Modifying existing practices
Rethink and renew practices
Transformative practices
Focus
Professional development
Professional learning
Learning process
Passive recipient of knowledge
Uncritical consumer
Collaboration
Mutual engagement and knowledge creation
Approaches
One-off seminars outside expert programmes
Programmes devised by expert over an extended time
Collaborative learning circles, networks, action research
Practitioner enquiry or action research, enquiry as stance
Professional outcomes
Improving teaching skills
Updated discipline knowledge or pedagogical skills
New approaches to pedagogy and learning
Production of new knowledge
View of teaching/ professionalism
Teacher as technician
Controlled
Teacher as craft worker
Compliant
Teacher as reflective learner
Collaborative
Autonomous professional
Activist