1. Learning about Learning – Evaluation of a National Child Protection Training Programme Kate Skinner Institute Lead: Research Development and Application

2. In this Presentation I will: Tell you about the training programme Describe our evaluation methods Identify what we learned from it Invite your thoughts and views

3. The Programme: Followed child death where knowledge of child protection (CP) found lacking in social work staff who worked with adults Funded by Scottish Government (formerly Scottish Executive) Available to social workers from 32 local authorities in Scotland

4. The Programme (ii) Aimed to include learning about substance misuse, domestic violence and mental health Comprised 4 days for adult services staff (2 days on CP and 2 days with staff from children’s services on joint working) 2 days for children’s services staff Delivered locally by project staff

5. Programme (iii) Programme accredited for 20 credits (200 hours study) at SCQF level 9 (3 rd year of a 4 year degree) with written assignment Hard copy materials not provided for participants tho’ virtual learning environment arranged for participants to access materials

6. The Evaluation (i) Commissioned via successful competitive tender Funded by the Project Commissioned in 2005 and completed by independent team from Universities of Stirling and Kingston in March 2007

7. The Evaluation (ii) Based on Kirkpatrick’s (1994) four levels of evaluation Used a multi-modal approach

8. Aims of the Study - To Evaluate impact on : Practitioners’ knowledge Intra-agency cooperation Intra-agency communication Initial assessments Ability to identify children at risk of harm Practitioners’ confidence re roles and responsibilities

9. Study Design Classroom Observation Scrutiny of Programme Materials Scrutiny of Participants’ Feedback Knowledge tests Short vignettes in which participants applied learning Trainers’ views on programme Scrutiny of assessment grades External Examiner’s Reports Participants’ views on changes to practice Survey of Managers Interviews with participants Interviews with service users Examination of service users’ files

10. What does the Literature tell us? (i) Evaluation must be systematic & include transfer of learning in the workplace (Baginsky and MacPherson, 2005; Ogilvie-Whyte, 2006) Collaborative working is difficult (Cooper et al, 2003; Huxham & Vangan, 2005) There is a knowledge base to be learned (Shardlow et al, 2004) Learning needs to connect to what people do (Rogers, 1974; Gardner, 2006)

11. What does the Literature tell us? (ii) Learning needs systematic preparation and support (Cherniss, 1998;Skinner & Whyte, 2004) Learning is shared responsibility of commissioners, learners, managers and trainers (Curry et al, 1994) Without involvement of all above, retention of learning and implementation will not occur systematically (Woodhouse and Pengelly; 1991 Fineman, 1997)

12. Findings (i) Little/no preparation of participants by managers or trainers Participants had v low expectations of programme as trigger for practice change Significant differences in delivery between project team members Disappointing changes in level of knowledge Major discrepancies between feedback and transfer of learning

13. Findings (ii) Some resistance to thorough evaluation of training as legitimate use of staff time Assessment of learning given v low priority by participants (3% of whole population) Self-report limited as measure for retention of learning Little attention given to retention of learning by staff, managers and trainers

14. Findings (iii) Intra- and inter-agency communication and collaboration is difficult and requires dedicated learning programmes to both raise their profile and enable learning of techniques

15. Concerns (i) Rhetoric of measurement, effectiveness and value for money not backed up in practice Self-reports viewed as sufficient proof of worth of training Absence of reliable objective data on impact

16. Concerns (ii) Suspicion that very little practice change resulted despite expensive, competent training arrangements Concern that government believe that training offers a speedy, reliable and productive response to a practice problem

17. Questions: Would it be better to do less training and focus more on retention? Are we using research on how people learn? Is it OK to go on a course and not expect to have to change what we do? Do we need to do more evaluation of this type to understand more about what kind of learning we should be offering?