1. Evaluation: A Necessity for the Profession, an Essential Support for Practitioners Linking Intervention with Outcome Bryan Hiebert University of Victoria University of Calgary 1

2. Overview: 4 main points Re-conceptualizing professional practice 1.Expand the scope of accountability 2.A framework for demonstrating value 3.What are the legitimate outcomes resulting from career development services 4.Some specific examples 2

3. Focus on Accountability Traditionally, focus was on outcomes Are the outcomes (results, effects, etc.) promised, achieved in practice? Need to broaden the focus … Are the outcomes achievable, realistic, and relevant? What factors contribute to the outcomes? What interventions make the outcomes more likely to be achieved? How do we know that the outcomes are a result of the interventions? 3

4. 4 Input  Process  Outcome Framework developed by the Canadian Research Working Group on Evidence-Based Practice in Career Development Outcome Focused Evidence-Based Practice Resources  What I Do  Effects of what I do 1.Comprehensive enough to include what is needed 2.Simple enough for people to use 3.The goal is to incorporate evaluation into service delivery

5. 5 Outcome Focused Evidence-Based Practice Indicators of client change 1. Learning outcomes 2. Personal attribute outcomes 3. Impact outcomes

6. 6 Outcome Focused Evidence-Based Practice Indicators of client change 1.Learning outcomes Knowledge and skills linked to intervention 2.Personal attribute outcomes Changes in attitudes, Intrapersonal variables (self-esteem, motivation, independence) 3.Impact outcomes Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training Societal and relational impact Economic impact

7. 7 Outcome Focused Evidence-Based Practice Resources  What I do  Effects Activities that link to outputs or deliverables Generic interventions Working alliance, microskills, etc. Specific interventions 1.Interventions used by service providers Skills used by service providers Home practice completed by clients 2.Programs offered by agency To reduce employment barriers To enhance client employability 3.Involvement by 3 rd parties 4.Quality of service indicators Stakeholder satisfaction

8. 8 Outcome Focused Evidence-Based Practice Activities that link to outputs or deliverables Specific interventions 1.Career decision making 2.Work-specific skills enhancement 3.Work search 4.Job maintenance 5.Career-related personal development 6.Other Resources  What I do  Effects

9. 9 Outcome Focused Evidence-Based Practice Resources  What I do  Effects Resources available 1.Staff Number of staff, level of training, type of training 2.Funding Budget 3.Service guidelines Agency mandate 4.Facilities 5.Infrastructure 6.Community resources

10. 10 Outcome Focused Evidence-Based Practice ResourcesClient change Knowledge Skill Attribute impact Counsellor Skills Interventions Programs Continuous Improvement Framework Quality Improvement of Services Resources  What I do  Effects

11. 11 Outcome-Focused Evidence-Based Practice Resources  What I do  Effects Intervention = Process + Outcome What will I do? + How is it working? Professional Practitioner

12. 12 Outcome Focused Evidence-Based Practice Input  Process  Outcome Need to link process with outcome 1.What will I do? 2.What are the expected client changes?  What client learning will result?  What sorts of personal attributes will the client acquire?  What will be the impact on their lives? 3.How will I tell?

13. 13 Evidence-based Practice 1.Research trials (traditional way in psychology) 2.Professional Practitioner  Purposeful intervention  Data (= evidence) on what was done  Evidence on client change  Look for patterns in data linking intervention with outcome  Develop scientific attitude toward practice –Skepticism, curiosity, inquiry A viable, perhaps even preferable, alternative scientific way to demonstrate effectiveness

14. Building cause and effect arguments What counts? (from initial CRWG survey) Career-life plan Increased self-confidence Greater capability to manage career-life plan Clients taking action as set out in their career-life plan Services delivered match client identified needs Use of resources How closely do service provider and client follow program How often is career-life plan modified Change in employment status Socio-economic impact  How do we document these? 14 Some of these require long term follow up

15. 15 Assessment as Decision Making (vs. Judgement) Please use a two-step process 1.Would you say that your level of mastery of the attribute under considerations is 2.Then assign the appropriate rating  0 = really quite poor  1 = just about OK, but not quite  2 = OK, but just barely  4 = really very good  3 = in between barely OK and really good acceptable unacceptable 0 123 4 4 0

16. Problem with skill self-assessment Participants asked to rate their skill (or knowledge) before and after a program Often, pre-workshop scores are high and post-workshop scores are lower  People find out as a result of the workshop that they knew less than they thought or had less skill than they thought  Based on the new awareness, post-scores are lower People don’t know what they don’t know How can we get around this problem?

17. 17 Assessing Learning & Attribute Outcomes Post-Pre Assessment We would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now? Please use a two-step process:  Decide whether the characteristic in question is acceptable or unacceptable, then  assign the appropriate rating acceptable unacceptable 0 123 4 4 0

18. Promising Practices 1.Several projects have been able (or will be able) to provide a solid link between programs & services and client change Provide two examples 2.Encourage you to think about how these ideas could be applied in your workplace 18

19. Starting Small: An Example In Progress Client characteristic  Dependable Indicators ( things that tell me a client is dependable )  Punctual  Follow through on a promise  Takes personal responsibility  Honest What I do to promote client dependability  Provide focus for client  Keep client on track  Encourage/reinforce client to follow through 19

20. Tracking Process and Outcome Maintaining Focus Not done Sort of done Done well Dependable Not really Sort of Defin itely  Provide focus for client   Punctual   Keep client on track   Follow through on a promise   Encourage client to follow through   Takes personal responsibility   Honest  20 A sample checklist OR use the 5 point decision-making scale described earlier

21. Applied Career Transitions Program (on-line program for unemployed university grads) For Module 1 All together there were 10 (items) x 29 (participants) = 290 ratings Pre: 144 Unacceptable Ratings – Post: 3 Unacceptable Ratings Unacceptable Ratings decreased from 50% to 14% Pre: 6 Exceptional Ratings – Post: 130 Exceptional Ratings Exceptional Ratings increased from 2 to 44% of the participants 21

22. Results: Impact outcomes Module 1  23 out of 29 had found a job  10 of the jobs lined up well with career vision Module 2  4 out of 6 had found a job  3 of the jobs lined up well with career vision In total  29 participants  27 found jobs  13 had jobs within their ideal career vision 22

23. 23 Attribution for Change To what extent would you say that the changes depicted above were the result of completing Module 1 of the ACT program, and to what extent were they a function of other factors in your life? mostly other factors somewhat other factors uncertain somewhat this program mostly this program  0001019

24. Linking Process and Outcome We have evidence that the facilitator followed the program We have evidence about which participants followed the program more (vs. less) closely  We could compare the results of those who follow the program compared to those who did not We have evidence of how much change took place We have a connection facilitator process  client engagement  outcomes Participants attribute change to the program A solid link between program and outcome Impact Learning Attribute 24

25. 25 Summary 1.Counsellors and agencies think it is important to evaluate their work with clients BUT they don’t do it 2.They need a framework for integrating evaluation and intervention What works and why 3.They need tools and resources for both short term and long term effects 4.We need an evaluation culture A professional identity that defines our role as BOTH delivering services & assessing outcomes