1 Outcomes of Social Work Education (OSWE): Findings from the Projects Hilary Burgess, John Carpenter, Joanna Fox, Roxana Anghel, Sharon Vitali, Juliet Koprowska, Kish Bhatti-Sinclair, Anne Quinney, Imogen Taylor, Clare Ockwell, Suzy Braye, Michelle Lefevre

2 Outline of Session 1. Introduction to OSWE project 2. Presentation of findings from six partners in England with questions for clarification. 3. Discussion of the findings in general and their implications. 4. Key issues in doing outcomes-focussed research on Social Work Education. 5. Open discussion.

3 Aims of OSWE (3 year project) To test the feasibility of outcome measures and research designs in Social Work Education To generate high quality evidence about the effectiveness of methods of SWE To build capacity and capability amongst academics, including service users To use opportunities to compare and contrast practice between programmes

4 A Collaborative Capacity & Capability building model Peer learning through Action Learning Set Support, advice and mentoring, F2F, and by phone (e.g. data analysis). See Burgess, Hilary and Carpenter, John (2008) 'Building Capacity and Capability for Evaluating the Outcomes of Social Work Education (the OSWE Project): Creating a Culture Change', Social Work Education.

5 Measuring Outcomes Measuring changes over time Before-after designs Using and adapting measures

6 Primary levels of Outcomes (after Carpenter, 2005) Levels of Outcome 1.Learners’ reactions 2.Modifications in attitudes and perceptions Attitudes Motivational 3.Acquisition of knowledge and skills Procedural Strategic knowledge Initial skills Compilation skills 4. Changes in Behaviour 5. Benefits to users and carers Focus Attitudes to race and racism (Soton) Attitudes interprofessional practice (Sussex) Understanding partnership with SU&C (Anglia Ruskin) Communication skills with children (Sussex) Interviewing and communication skills (York) Use of research skills (Bournemouth) Acquisition of Soc. Wk. competences (Oxford Brookes)

7 The outcomes of teaching and learning about ‘race’ and racism Kish Bhatti-Sinclair Division of Social Work Studies School of Social Sciences University of Southampton

8 Aims and Methods Students’ understanding and experience of ‘race’ and racism; and How ‘race’ and racism is addressed in the degree curriculum Questionnaires to BSc social science students in week 2 of Semester 1 (153 students) and in Week 8 (71 students) of Semester respondents completed both sets. Social work students = 41.2%.

9 Main sources of learning about race and racism (frequency) TYPES OF LEARNING Seminars 17 Course units16 Personal study 9 Shadowing soc wkr 4 Peer Group 3 Personal tutors 2 SPECIFIC COURSE UNITS 2 out of total of 8 (3 SW) Social problems and social policy - 6 Sociology of everyday life - 2

10 Results A small increase in self-rated knowledge of racism – 10%. A increase in respondents’ confidence in challenging racism (but not statistically significant.) Overall results suggest that formal teaching about ‘race’ and racism did influence knowledge although the impact was small.

11 Limitations Sampling – not random. Results not generalisable – must be interpreted with caution. Knowledge of racism based on self rating - not possible to obtain an objective measure of knowledge.

12 Outcomes of “Using Research for Practice” using blended learning. Bournemouth University Anne Quinney BA Social Work Year 2 students.

13 Methodology Pre-Post tests Measure: Research Self Efficacy (RSE) scale (Holden et al., 1999). Subscales on research knowledge and skills. PLUS 5 items “using computer and information technology”. Analysis: Paired t-tests for difference in mean scores.

14 Findings: Research Knowledge and Skills “How confident are you that you can successfully analyse basic quantitative and qualitative data?” Scale 0-10

15 Findings: using computer and information technology “How confident are you that you can successfully access research findings from research bodies, social work organisations, government departments etc (e.g. JRF, SCIE) using the internet?”

16 Next Steps and Uses 1. Do Research Self Efficacy scores and assignment marks correlate? 2. Compare with another programme Uses: – Adjust the curriculum for (1) present students who have to do a dissertation in year 3 and (2) next year’s course. – Student self-assessment: “what I need to learn.” – The RSE scale has been adopted by ESRC Researcher Development Initiative.

17 Outcomes of teaching about Partnership & Interprofessional Practice Imogen Taylor, Clare Ockwell, Suzy Braye University of Sussex Note: Class-room based module does not include students from other professions.

18 Methodology Outcomes of learning and teaching on BA & MA students’ attitudes – Stage 1 pre module teaching – Stage 2 post module teaching – [Stage 3 end of course] Validated scales from University of West of England IPE programme – Communication and teamwork (CT) – Interprofessional learning (ILL) – Interprofessional interaction (II) – Interprofessional relationships (IR)

19 Mean ratings at start/end of module (BA)

20 Differences in attitudes by years of pre-course experience

21 Next steps Complete analysis of BA/MA T1 and T2 Collect and analyse end programme BA/MA data (T3) Compare with UWE IPE findings Introduce new comparator programme (any offers?)

22 Outcomes of Teaching and Learning communication skills Juliet Koprowska University of York

23 Research question & methodology Are communication skills improved through learning within the university? Methodology: multiple-measure case study design. Sample: year 1 BSc Social Work

24 Research methods Self-efficacy scale Recorded interview with service user-actor Self-evaluation of interview Service user-actor evaluation Assessment of video-recording Feedback from service users on placement T1 = prior to teaching, T2 = 3 months later, after teaching, T3 = 9 months later, after first placement

25

26

27 Implications Students may lose confidence as they acquire knowledge and understanding Potential for formative assessment Effect of placement needs exploration Qualitative discussion with students and actors would deepen understanding Should tools become regular measures to increase participation in research, with retrospective permission?

28 Communication Skills with Children & Young People Michelle LeFevre University of Sussex How does a Programme contribute to students’ development of confidence and competence in communicating with children and young people?

29 What and How are these learned? ‘Skilled/effective communication’ is not just techniques and micro skills. Also underpinning knowledge, values and ethical commitments, personal qualities and emotional capabilities. Learned throughout the whole programme not just in focused skills teaching – and also through other personal and professional experiences.

30 Methodology Prospective Evaluation – 4 stages 1. Questionnaire – Students’ personal characteristics: Have particular kinds of student learned most/least? – Measuring self-confidence in communication with children at different stages. – Which aspects of the programme facilitated your confidence and skills? 2. Case Vignette tool – Students demonstrate knowledge of planning, implementing, reviewing and reflecting on communication with children. – More objective measure (though analysis is subjective?)

31 Key Findings 1. Students’ confidence has increased. 2. Direct practice in placement the most important, followed by Child Development 3. All aspects of focused skills teaching were perceived as helpful to learning 4. Knowledge of aspects of communication skills increased (often significantly) - but not in every domain. 5. Still to analyse relation of characteristics to learning and mapping individual students through programme.

32 Learning about working in partnership with service users Roxana Anghel & Joanna Fox Anglia Ruskin University Exploring the use of Concept Mapping

33 “ Schematic representation of an individual ’ s understanding of a knowledge domain in the form of concepts meaningfully linked in propositions. ”  Design – CM and questionnaire at T1 (induction week BA); T2 (end year 1); T3 (end year 2) – CM task, unstructured - ‘Working in Partnership with Service Users’ – Questionnaire – most impacting learning opportunities. Analysis of CM – Generate scores based on validated concept-links – Raters included service user and researcher. Method – Concept Mapping (CM)

34 Concept Map at start of programme

35 Concept map at end of Year 1

36 Concept Map at end Year 2

37 Findings – “Working in Partnership” Map Scores

38 Sources of Learning At T1: Ethics and Values; observational practice; Poverty, Social Exclusion and Social Work, and academic reading. At T2: Social Work with Children and Families, Social Work with Adults, practice placement; Principles and Skills of Social Work; academic reading; and discussing with colleagues

39 CM used on two other modules Module B – Map scores and marks did not correlate Module A – Map scores correlated with marks Module B (Year 1 MA, 11 pairs) – 72% T2 maps increased content and quality Module A (Year 3 BA, 13 pairs) – 46% T2 maps increased content and quality Students Map Scores Students Map Scores

40 Discussion – Small numbers, so the value of CM as summative method in assessing outcomes needs to be explored further – Success depends on the clarity of instructions and purpose, familiarity and time available. – Uses: self-assessment, visual learning aid, formative assessment (identify gaps), as well as assessing outcomes. – Being firmly based on “meaningful” learning CM can be applied to many area of social work.

41 Measuring the Acquisition of Competency Sharon Vitali Oxford Brookes University

42 Methodology Cohort Sample BSc Social Work – (n=38 at beginning n=34 at present = 89%) Longitudinal - Baseline to Graduation Repeated Measures Fixed Outcomes (National Occupational Standards Units) RICET Multiple Methods

43 Year One - Semester One Baseline December RICET PERSONAL LEARNING PLAN

44 Consider your current state of competence (knowledge/skills) and indicate a score (from the guide below) for each of the learning outcomes listed. 0 = Can not produce any evidence of competence. 1 = Understands the learning outcome, but can produce only limited or no evidence of appropriate attempts to put it into practice. Much more knowledge/practice needed. 2 = Understands, and can offer evidence of tentative attempts to integrate into current knowledge/skill base. 3 = Demonstrates competence with some regularity. 4 = Advanced understanding and demonstrating adequate level of integration of knowledge, skills, and appropriate application. 5 = Clearly understands and demonstrates consistent and appropriate application of knowledge and skills in practice.

45 Academic Average = 61.4% Video Scores SU=1.8 ER=2.2

46 Year Three - Semester Two May (70 DAY PLO) RICET PRACTICE & PORTFOLIO TUTOR EVALUATION PRACTICE & PORTFOLIO P/T EVALUATION CRITICAL CAREER REVIEW TUTOR EVALUATION RICET

47 PA End Place2 ST End Place2 PA Mid Place2 ST Mid Place 2 PA End Place1 ST End Place1 PA Mid Place1 ST Mid Place1 ST End Yr1 S2 Tutor End Yr1S1 ST End Yr1 S1 ST Baseline

48 Types of Data Y1- S1 RICET numeric LEARNING PLAN p/f RICET numeric Grades numeric Y1- S2 INTERVIEW numeric RICET numeric PORTFOLIO p/f INTERVIEW numeric Grades numeric Y2- S1 RICET numeric RICET & PORTFOLIO Numeric & p/f PORTFOLIO p/f Grades numeric Y2- S2 RICET numeric RICET & PORTFOLIO Numeric & p/f PORTFOLIO numeric Grades numeric Y3- S1 RICET numeric RICET & PORTFOLIO Numeric & p/f PORTFOLIO p/f CASE VIVA numeric Grades numeric Y3- S2 RICET numeric RICET & PORTFOLIO Numeric & p/f PORTFOLIO numeric CRITICAL CAREER REVIEW numeric Grades numeric NOW end

49 Overview of the findings from projects increases in: 1. Attitudes to race and racism - but not to self- confidence in challenging racism. 2. Self-efficacy following learning (e.g. research skills) Do these correlate with marks? 3. Conceptual understanding (C Maps, communication vignettes). 4. Ratings of Competence (RICET)

50 But: 1. No measured increase in Communication skills 2. No overall increases in attitudes to partnership working 3. Measures do not necessarily correlate with marks. And we need to strengthen research designs, especially comparative studies.

51 Some Questions for Discussion 1. What methods of learning and teaching should we compare? Are you interested? 2. What is their potential for use in formative evaluation and self-evaluation by students? 3. Should we use these measures routinely (like Oxford Brooks) to assess student outcomes? 4. Are you interested in replicating and developing these methods?

52 Key issues 1. Formulating an appropriate, specific research question 2. Selecting/adapting appropriate methods/measures for measuring change 3. Negotiating student engagement, balancing ethical concern to allow opt-out with maximising engagement to promote learning 4. Establishing involvement of service users/carers 5. Getting support from colleagues 6. Freeing space/time/resources for the research

53 Key issues (cont) 7. Balancing roles of teacher and researcher (sometimes complementary, sometimes conflicting) 8. Accessing expertise/learning to analyse and interpret data 9. Finding comparison sites 10. Feedback to programme/colleagues about knowledge/skills gained and implications for rest of programme

54 Thank you To our sponsors: SCIE, SWAP and IRISS Contacts: Resource: Evaluating Outcomes in Social Work Education