January, 2014

 National research agenda to better understand “what works” in career services  3 calls for proposals from HRSDC over 6 years  Source of major international envy among Career Development Practitioners (CDPs) and Researchers Evidence-Based Research in Canada 2

 Ethics: Clients deserve interventions that have demonstrated success  Confidence: Practitioners should know that a given intervention will result in a predictable outcome  Security: Being able to provide evidence of success increases ability to obtain and keep funding for services and secure jobs for CDPs/Employment Counsellors Why Evidence-Based Practice? 3

 Outcome-focused intervention  Evidence of client change …contrast with…  Client flow  Ease of access  Practitioner time use It’s easy to measure variables that don’t address client change Why Evidence-Based Practice? 4

5

 Highlight results of 3 recent research projects: What have we learned?  Explore how a select number of LMI and career development resources could be integrated into everyday evidence-based practice  Explain where we are now and next steps  Direct you to additional research studies Session Objectives 6

 What evidence do you currently collect?  What is done with the evidence currently collected? 7 Before we start…….

 What did the research do?  What was learned?  What was most surprising?  What are the implications for practice? Reflective Questions…. 8

R ESEARCH P ROJECT #1: A SSESSING THE I MPACT OF LMI ON C AREER D ECISION M AKING AND W ORK S EARCH Research Team Canadian Research Working Group in Evidence- Based Practice (CRWG), Canadian Career Development Foundation (CCDF) In partnership with New Brunswick Post-Secondary Education & Labour, Saskatchewan Advanced Education, Employment & Labour 9

WHAT IS THE PROBLEM?  There is very little literature about the direct contribution of LMI to employment/career decision-making ( Savard & Michaud, The Impact of LMI on Career Decision-Making Process: Literature Review, FLMM, 2005)  Several questions remain unanswered:  How do people actually use LMI?  What (if any ) assistance would be helpful? 10

RESEARCH QUESTIONS  If client needs are assessed and clients are given LMI consistent with their needs,  To what extent does assistance by a service provider enhance their effective use of LMI? OR  To what extent is independent self-help a sufficient process for clients to use LMI effectively? 11

METHOD  We prepared “guided” LMI packages (take- home booklets and Resource Centre binders) on:  Career Decision Making:  Know yourself  Know the Labour Market  Put it all Together  Job Search:  Check for “Fit”  Get Ready  Search for Work  Get a Job 12

All participants in the study:  Received a needs assessment interview & completed an initial, pre-program survey  Received an LMI package specific to their identified employability need  Were randomly assigned to either a self- directed intervention delivery method or an assisted self-directed intervention delivery method.  Were given an orientation to the Resource Centre which they could freely use on their own  Completed a pre and post-pre survey  Received a cash honorarium and certificate of participation 13 INTERVENTION

The self-directed group:  worked independently for 3 weeks, making use of the materials and the Resource Centre  returned in Week 4 for their exit interview 14 INTERVENTION (CONT)

The assisted self-help clients received:  two additional AIS (Advice and Information) interviews (20-30 minutes) in weeks 1 and 3 focused on helping them understand, interpret and apply the LMI to their own situations and /or access additional LMI  returned in Week 4 for their exit interview 15

SAMPLE BY PROVINCE ProvinceIntervention Type Delivery Total IndependentAssisted Saskatchewan CDM JS Province Total New Brunswick English CDM JS Province total New Brunswick French CDM JS11920 Province total26 52 Total for 3 groups

WHAT DID WE MEASURE The Dependent Measures for the data analysis were:  General ability to use LMI  Knowledge  Clear vision of what I want in my career future  Knowledge of print and online resources  Skill  Have effective strategies for keeping myself motivated  Have a realistic action plan  Personal Attributes  Optimism about what lies ahead re meeting my career goals  Confidence in my ability to manage future career transitions 17

EXAMPLES OF QUESTIONS USED IN PRE AND IN POST- PRE SURVEYS Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now? Before Av e After Ave 1.A clear understanding of what I need to do to move forward in my career A clear vision of what I want in my career future Reviewed my past work, education and experience so that I know what skills and strengths I have A list of possible options that may fit with what I want in my career future Knowledge of print and online resources that help me to research career/employment options Confidence that career-related employment opportunities actually exist that fit with what I want in my career future

 Changes in competency self assessments: Pre program:  30% and 58% of the respondents rated competency Not OK  5% rated their competency as Exceptional Post program:  1%– 9% rated competency Not OK  39% rated their competency as Exceptional The amount of change was similar across all three dimensions of the survey, knowledge, skills, and personal attributes all demonstrated about the same amount of change. 19 DESCRIPTIVE RESULTS

DESCRIPTIVE RESULTS (CONT.)  Of particular note are items that suggest increased ability to self- manage their careers, such as:  A clear understanding of what I need to do to move forward in my career.  A clear vision of what I want in my career future.  Effective strategies for keeping myself motivated to achieve my career/employment goals.  Confidence in my ability to research career, employment, and training options that are available  Confidence in my ability to manage future career transitions. 20

DIFFERENTIAL RESULTS-TOTAL SCORE  Both CDM and JS groups had significant increases across time  Change in CDM group was significantly larger than in JS group  Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began.  Participants receiving assistance demonstrated greater change across time than did those in the independent mode  Participants receiving assistance demonstrated greater change across time than did those in the independent mode Similar pattern for all subscales 21 For group as a whole: significant increase in overall ability to use LMI significant increase in overall ability to use LMI neither intervention was more conducive to one manner of delivery compared to the other neither intervention was more conducive to one manner of delivery compared to the other

To what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life? 22 mostly other factors somewhat other factors uncertain somewhat this program mostly this program English French Program ATTRIBUTION FOR CHANGE

 Overall,no significant differences in the results of the anglophone and francophone samples with the exception that assisted francophone clients reported increased gains over the anglophone clients although both groups reported more positive change in the assisted condition. 23 STUDY REPEATED IN FRANCOPHONE NEW BRUNSWICK

FOLLOW UP INTERVIEWS (1 WEEK AND 4 MONTHS AFTER INTERVENTION)  “I was surprised at how much information there was”  “Initially I thought I could not do this, but I found out I could”  “There’s a lot to learn”  “I got more focused on my goal”  “I got clearer about what I wanted to do” 24

CLIENT CRITICISMS  The program needed to be longer given the number of resources available. (222)  Working full time made it very hard […]. With a little more help it would have been much easier (242)  There was too much information and that made making a career decision more difficult […] very hard to do this alone (209) 25

WHAT DID PRACTITIONERS SAY?  “The structure helped me be more focused and concrete”  “I never really thought about intake of clients as assisted or self- help—now it’s built into every intake I do”  “I used to be maternal and forget that it was the client’s decision and I got more involved than I needed to. Now I ask the clients to be more responsible”  “Now I give more homework tasks and I’m more specific. The clients felt more focused and so do I” 26

WHAT DID THE RESEARCH DO?  Isolated LMI as an intervention  Isolated giving information and advice from employment counselling  Organized LMI into a career development learning sequence  Tested self-help versus a combination of self- help, information giving and advice 27

WHAT WAS LEARNED?  Tailored LMI embedded in a learning process results in knowledge and skill acquisition as well as the capacity for self-management  LMI appropriate for a client’s specific need (opposed to general LMI) appears to support engagement and action  For many clients, a little (or no) professional LMI support is enough  Structure and timelines appear to motivate action and a sense of progress  Giving clients hands-on tools appears to motivate more than money 28

WHAT WAS MOST SURPRISING  The extent of positive client change in knowledge, skill and personal attributes in both conditions of self-help and supported self-help  The value clients placed on the LMI materials and the degree to which they used them  The positive impact of “organized” LMI and LMI “specific to client need” versus generalized LMI 29

WHAT ARE THE IMPLICATIONS FOR PRACTICE?  To be discussed…….. 30

 Drs. Dave Redekopp and Bryan Hiebert  Alberta Human Services  Manitoba Entrepreneurship, Training and Trade  Canadian Career Development Foundation (CCDF) Research Project #2: Assessing the Impact of Career Resource and Supports Across the Employability Dimensions 31

 There is very limited research about the direct contribution of career development learning resources on client progress toward or achievement of employability.  There are many “assumptions” made about self-help resources but very little actual evidence of their impact.  There is much “talk” about labour market attachment but on investigation it remains a very “wooly” construct. 32 What is the Problem?

Developed comprehensive career development coaching guides (way beyond LMI) covering all employability dimensions excepting career growth:  CAREER DECISION MAKING: Not yet clear about work/learning goal; needs to understand personal skills, interest, values and personality and more about different potential occupations in order to develop a vision for a preferred future  SKILLS ENHANCEMENT: Has career goal, but needs to explore skills and qualifications needed for the work they want and how to acquire them  JOB SEARCH: Has career goal and necessary skills/qualifications for desired work; needs skills, strategies and tools necessary to conduct a successful job search  JOB MAINTENANCE: Has difficulty keeping work; may need to address personal/life issues (e.g. health, behaviour, transportation, child care) or may need to work on the skills/attitudes required for success at work. The last part of this Guide is appropriate for someone already working, but wanting to improve their chances of keeping the job. 33 This Study…

 Randomly assigned clients to one of two treatment conditions:  Practitioner launched  Practitioner launched (in which they had their needs assessed, were oriented to the appropriate Guide and then worked independently with their Resource Guide for 4 weeks)  Practitioner launched and supported  Practitioner launched and supported (in which they also worked with their Career Consultant for 4 weeks of “regular” services (i.e. 2-4 sessions on average) using the Guide as appropriate but not restricted in their interventions  Examined differential impact across treatment conditions on client knowledge, skills, personal attributes and labour market outcomes 34 This Study…

Self Help Index(self-help-o-meter) to see if:  The propensity for self-help can be measured  The ability to self-help affects client outcomes Labour Market Attachment Index to see if:  Specific variables related to the client’s work history and life situation can be measured  These variables affect client outcomes Developed Two New Indices…

If clients are given a comprehensive needs assessment to determine their employability need(s), what is the differential effect of independent and consultant-supported career development resources on clients who are weakly attached to the labour market versus those who are strongly attached to the labour market?  Will all clients demonstrate positive change?  Will clients who are strongly attached to the LM show more positive change?  Will clients who are supported by a consultant show more positive change?  Will weakly attached clients struggle more in the independent condition? 36 Research Question

ProvinceDimensionDelivery Alberta = 115CDM = 35Independent = 17 Supported = 18 SE = 25Independent = 16 Supported = 9 JS = 42Independent = 21 Supported = 21 JM = 13Independent = 10 Supported = 3 Manitoba = 113CDM = 44Independent = 21 Supported = 23 SE = 19Independent = 10 Supported = 9 JS = 43Independent = 23 Supported = 20 JM = 7Independent = 4 Supported = 3 Totals Total sample = 228 CDM = 79 (35%) SE = 44 (19%) JS = 85 (37%) JM = 20 (9%) Independent = 122 (54%) Supported = 106 (46%) 37 Sample Composition

DeliverynBeforeAfterTotal PLR (11.38)45.50(10.93)34.60 PLSR (11.41)49.05(8.86)36.99 Total (11.98)47.38(9.98) Key Findings Career Decision Making Total Score Summary: Main Effect for Delivery: F(1,70)=1.32; p=.26 Main Effect for Time: F(1,70)=253.72; p<.01 Delivery X Time Interaction: F(1,70)=.66; p=.42

DeliverynBeforeAfterTotal PLR (21.50)93.95(8.05)77.67 PLSR (18.75)88.88(17.61)69.53 Total (20.85)91.76(13.10) Key Findings Skill Enhancement Total Score Summary: Main Effect for Delivery: F(1,35)=.48; p=.04 Main Effect for Time: F(1,35)=69.33; p<.01 Delivery X Time Interaction: F(1,35)=.51; p=.48

DeliverynBeforeAfterTotal PLR (23.76)97.46(13.52)78.99 PLSR (23.24)97.97(12.69)75.88 Total (23.61)9769(13.07) Key Findings Job Search Total Score Summary: Main Effect for Delivery: F(1,72)=.71; p=.40 Main Effect for Time: F(1,72)=260.49; p<.01 Delivery X Time Interaction: F(1,72)=2.07; p=.16

DeliverynBeforeAfterTotal PLR (12.30)73.50(9.70)64.58 PLSR556.20(18.38)77.80(14.96)67.00 Total (13.74)74.94(11.14) Key Findings Job Maintenance Total Score Summary: Main Effect for Delivery: F(1,15)=.18; p=.68 Main Effect for Time: F(1,15)=28.03; p<.01 Delivery X Time Interaction: F(1,15)=.19; p=.67

 91% of clients in all interventions attributed their positive change partly (40%) or mostly (51%) to their participation in the study  92% had a clear plan for next steps after the 4 week intervention period  98% planned to use their Guide again and/or shared it with friends and family 42 Attribution of Change

 The Guide was hugely beneficial. It was a profound and amazing experience.  My current job is not a good fit for me, but the job I have just been offered (after working with the CDM guide) is the PERFECT fit for me.  The guide really clarified for me what I want to do in my work – what is important to me. It helped me to realize I have the qualifications and [know]what I love to do. It re-connected me with myself.  The guide was a catalyst for action. I knew in the back of my mind for a long time that I wasn’t happy in my work, but it was the guide that made me do something about it.  I think the guide is pretty universal for Canada. Combined with the skills I got from my employment counselor, I feel more confident and feeling like I’m going in the right direction. It was only 4 weeks but sometimes that’s all you need!  Definitely not feeling the way I did before the guide.  This experience was phenomenal. I’m so glad I got to be part of it! I’m grinning ear to ear!!  I think high school students should get this. 43 In the Words of Clients…

 Statistically and clinically significant positive impact of career interventions across ALL sub- scores and across ALL employability dimensions – compelling evidence of positive changes in clients as a result of career development coaching interventions 44 Results

 More positive trends for supported group but differences are not statistically significant ( surprising and troubling result until overly caring practitioners admitted to “creaming” the sample with the best of intentions and researchers realized too late that practitioners really do care!!) 45 Results (cont’d)

 Developed and tested impacts of career development coaching resources on changes in specific employability dimensions  Simulated regular employment services to the extent possible  Attempted to “unpack” two poorly understood constructs—capacity to self-help and labour market attachment 46 What Did the Research Do?

 Many clients can benefit significantly from self- help resources when they are matched to their need and they are “launched” – more so than we anticipated!  This is especially noteworthy given the short intervention period of 4 weeks and the “real- life” setting in which the changes occurred. 47 What We Learned

 Self-help coaching guides provided after a thorough needs assessment and orientation are effective and can be used as a first line of intervention, saving valuable practitioner time for clients who really need it, or potentially opening a promising perspective on online career services.  More research is needed to test the “Self-Help Index” and a “Labour Market Attachment Index”. These tools could not be effectively tested due to the unanticipated “creaming” of clients who, in the view of the practitioners, might have had difficulty with self-help materials (if assigned to the independent group) and/or might fall at the low extreme of labour market attachment. 48 What We Learned (cont’d)

 The statistically and clinically significant amount of change experienced by clients – especially given the short intervention period  The number of clients interviewed after the study who credited the guides for significant positive life changes  The extent to which practitioners remained committed to the research despite significant additional demands on their time 49 What Was Most Surprising?

What Are the Implications for Practice?  To be discussed…….. 50

Transforming the Culture of Evaluation in Career and Employment Services Saskatchewan Abilities Council Saskatchewan Department of the Economy New Brunswick Post-Secondary Training, Education and Labour Goss-Gilroy Inc. Canadian Research Working Group on Evidence Based Practice in Career Development (CRWG) Life Role Development Group (LRDG) CCDF 51 Research Project #3: Common Indicators

 What “inputs”/ “precursors” impact/influence client change?  What “processes”/ “interventions” impact/influence client change?  What “outcomes” result from the interactions between “inputs” and “processes”? 52 What Have We Learned So Far About Client Change?

53 Research Progress to Date Input  Process  Outcome Indicators of client change 1.Learning outcomes Ability to use LMI Ability to use career coaching resources Employment/career vision Strategies to stay motivated Skills across all employability dimensions 2.Personal attribute outcomes Changes in optimism and confidence 3.Impact outcomes Impact of #1 & #2 on client’s life, e.g., employment status, employment retention, training status, fit with goal and vision LMI Booklets AIS interviews Employability Dimensions Resource Packages Practitioner coaching using Resource Packages Resources available Self-help Index Labour Market Attachment Index

 We have data/evidence of positive impacts from specific interventions for certain kinds of clients  We do not have data/evidence of positive impacts from employment services overall  We do not have a data gathering tool that can be used to gather common data across divergent employment service settings 54 What is the Problem?

 Key research questions:  What common indicators are applicable across different contexts, different client groups, different agencies and different interventions?  What statements of service effectiveness can be made by tracking common indicators? and 55 Therefore the Next Step: Common Indicators Project

 What kinds of interventions in what contexts produce what kinds of results? 56 IF Numbers Permit… the Ultimate Question:

 New Brunswick Post-Secondary Training, Education and Labour (30 practitioners; 137 clients)  Saskatchewan Abilities Council (18 practitioners; 154 clients) 57 Participants New BrunswickSaskatchewan Campbellton/Bathurst Dieppe/Moncton Fredericton Miramichi Péninsule acadienne Edmunston Southwest Southeast Moose Jaw Regina Saskatoon Swift Current Yorkton

 Literature Review  Focus Groups  Compare the two  Accept those in common and most robust 58 What Common Indicators are Applicable? How we Investigated

 If clients have benefitted from career and employment services, they have experienced some changes in in their lives. They are not in the same position they were when they started. Some changes may be obvious—they have a job which they did not have before; they have enrolled in a training program which they had not before for example. What else indicates that clients or the situations of clients have changed as a result of career and employment services?  What clues do you look for to tell you that clients and/or client situations are changing? 59 Focus Group Questions: Indicators of Client Change: (Outcomes)

 What do you do or what do you provide that you think contributes to these changes in the lives of clients? 60 Contributors to Client Change: (Process)

 What do you think influences your ability to help clients and/or their situations to change?  What do you think influences the ability of clients to change themselves or their situations?  What external factors (not in the control of clients or staff) influence the changes achieved through career and employment services? 61 Influences on Staff, Clients/External: (Inputs)

62 PRIME Structure (Performance Recording Instrument for Meaningful Evaluation) Input  Process  Outcome Indicators of client change 1.Learning outcomes Changes in Knowledge and Skills Changes in Personal attributes 2.Impact outcomes Impact of #1 & #2 on client’s life, e.g., employment status, training status, quality, fit, standard of living 3.Working Alliance and Client engagement 4.Actions underway to move forward: Work in Progress Working alliance, client engagement Specific interventions in: Job Readiness Career Decision Making Skill Enhancement Work Search Work Maintenance Life Circumstances Goal Setting, Action Planning, Action Steps Support provided in personal attributes: self-esteem, self efficacy, well-being, self confidence, self- awareness Progress ratings (+ or -) in all of the above Resources available Practitioners: Background; Experience Availability of and access to Community resources Availability of and access to Employment Opportunities and Training Client Employment History and Potential Client Life Circumstances

 Recruit clients into the study (NO criteria);  Follow all research protocols  Use PRIME each time you see a client to assess and record any changes (+ or -) in:  Goals and Action Plans  Employability Learning Needs/Changes  Personal Attribute Needs/Changes  Life Circumstance Needs/Changes  Working Alliance and Client Engagement 63 Practitioner/Research Partner Tasks

 Work with each client for 6 weeks following Assessment interview and client entry into the research  Conduct an Exit from Research interview and use the system to record/assess:  Goals and Action Plans  Employability Learning Needs/Changes  Personal Attribute Needs/Changes  Life Circumstance Needs/Changes  Working Alliance and Client Engagement….AND 64 Practitioner/Research Partner Tasks

 Labour Market Outcomes:  Employment or Training Status  Fit of Employment or Training with skills, qualifications, vision  Adequacy of salary  Linked to future opportunity  Actions Underway to move forward: Work in Progress  Have clients complete post-pre questionnaire 65 Practitioner/Research Partner Tasks

Client needs assistance to : Not at all Not much A little Quite a lot A lot  Identify and clarify future direction (e.g., training, education, employment or change in life circumstances goal)  Identify personal strengths/resources that support future direction (e.g., training, education, employment or change in life circumstances goal)  Resolve specific challenges/vulnerabilities that may impact on future direction (e.g., mortgage, public transit, day care etc.) 66 Pre-Employability/Job Readiness

Indicators of challenges in client life circumstances are in evidence: Not at all Not much A little Quite a lot A lot  Improved housing is needed  Improved transportation is needed  Improved capacity to work and/or study is needed  Increase in sense of responsibility for own choices and behaviours is needed  Increase in ability to set short and long term goals is needed  Increase in understanding expectations and demands of employers is needed  Reduction in destructive behavior is needed  Following medical and medication protocols is needed  Increased openness to change is needed  Increased access to constructive and positive support systems is needed  Improved relationships with family and friends are needed  Increased trust in other people is needed  Other: _________________________________ 67 Life Circumstances

68 Personal Attributes Example Client needs assistance in the following areas: Not at all Not much A little Quite a lot A lot Ability to Self-Manage:  More info Developing stronger Self-Esteem:  More info Improving sense of Well-Being: i.e.:  More info Developing stronger Self-Efficacy: i.e.:  More info Increasing Self-Awareness:  More info  Other (Please specify): ________________________________

 more info: Self-Esteem: “a person’s overall emotional evaluation of his or her own worth, an attitude to self and a judgement of oneself. Self- esteem encompasses beliefs (e.g., I am competent; I am worthy) and emotions (pride, shame). Self-esteem is the positive or negative evaluation of the self”  Demonstrates an optimistic outlook  Sees self as competent and able  Has confidence in ability to interact with others  Uses positive self-talk  Acts with little hesitation  Raises few objections  Asks questions  Articulates skills assertively  Makes eye contact  Is hopeful  Takes reasonable risks 69 Self-Esteem Attribute

70 Working Alliance The client and I: Not at all Not much A little Quite a lot A lot  established a climate of trust and comfort in working together  arrived at a goal that is owned by the client  agreed on the action plan steps to help achieve the client’s goal

71 Client Engagement: The client: Not at all Not much A little Quite a lot A lot  participated actively in the interview  was focused on achieving results

 Of 154 clients for whom practitioners rated client personal attribute needs before and after, 40% improved, 14% worsened and 46% experiencing no change. 72 How Much Change in Personal Attributes (e.g., self-efficacy and self-esteem) Occurs For Clients Within 6 Weeks Of Intervention?

73 Contrast with Clients

74 How Much Do Clients Learn in 6 Weeks (Regardless of Employability Dimension)? Practitioners:

75 How Much Do Clients Learn in 6 Weeks (Regardless of Employability Dimension)? Clients:

Learning Change Low Alliance/ Engagement (# of Clients) Moderate Alliance/ Engagement (# of Clients) High Alliance/ Engagement (# of Clients) Improved2 (50%)12 (48%)81 (62%) Unchanged1 (25%)7 (28%)28 (21%) Worsened1 (25%)6 (24%)22 (18%) 76 How Does the Working Alliance Affect Learning Outcomes? Practitioner

Learning Change Low Alliance/ Engagement (# of Clients) Moderate Alliance/ Engagement (# of Clients) High Alliance/ Engagement (# of Clients) Improved4 (100%)20 (95%)243 (95%) Unchanged0 (0%)1 (5%)4 (2%) Worsened0 (0%) 8 (3%) 77 How Does the Working Alliance Affect Learning Outcomes? Clients

Overall, 31% of the 313 clients for whom outcome data exist were employed, 17% were in a training/education program and 7% were waitlisted for a program by the end of the intervention period. Where 65% of clients were not employed, in a program or on a waitlist before the intervention, 45% were after the intervention period, a decrease of almost one-third. 78 What Labour Market Outcomes Are Achieved with 6 Weeks of Intervention?

 Almost all clients (90%) whose learning improved also improved their personal attributes, in practitioners’ eyes. At the diagonal end of the table, the majority (65%) of clients whose attributes worsened also experienced decreases in learning.  Effectively all clients (99%) experiencing improved attributes also experienced improved learning. 79 How Are Improvements in Learning Outcomes Related to Improvements in Personal Attribute Outcomes? Improved Attributes Unchanged Attributes Worsened Attributes Improved Learning54 (90%)29 (45%)4 (20%) Unchanged Learning1 (2%)26 (41%)3 (15%) Worsened Learning5 (8%)9 (14%)13 (65%)

The following items were shown in this study to have statistically significant connections to employment outcomes: 80 Question #1: Applicable Common Indicators Input IndicatorsProcess Indicators Employability Dimension need/ competence (composite & 5 dimension scores) Responsibility Support Personal attributes Working alliance/ client engagement

 Working alliance/client engagement is a predictor of both employment outcomes and learning (as measured by reductions in need for help with employability dimensions) and is a likely predictor of improvements in personal attributes (e.g., self- esteem, self-efficacy)  Over ¾ of clients who obtain employment report finding employment consistent with their skills and vision  (N.B. 70% of sample had not exited services by end of study) 81 Question #2: Connecting Inputs, Processes and Outcomes

 The more clients need assistance with Skill Enhancement, the less likely that they will be employed in 6 weeks  The stronger clients perceive their personal attributes, career decision making and work search skills the more likely they will be employed in 6 weeks  Personal attributes improve over 6 weeks  Learning improves over 6 weeks  Client learning is highly related to client change in personal attributes 82

 The ability to assume personal responsibility and to have access to strong support systems appear to be among the most influential life circumstances with respect to outcomes  Practitioners were overall cautious in reporting client change; clients were not. More study is needed but it is clear that assessing change is complex and subtle—it is not a linear process with simple cause and effect connections 83

Model is robust but can be simplified and improved:  The number of indicators under each employability dimension can be significantly reduced and give the same result  Only two indicators under Life Circumstances emerged as significant—the ability to assume personal responsibility and have access to strong support systems. These indicators can be significantly reduced  Working Alliance and Client Engagement can be combined as one indicator only  A number of technical improvements to the system have been identified 84 Other Findings :

 We can collect common indicators across service settings, essential for the field to develop convincing evidence based data (still need larger samples and longer data collection)  We have a pathway to measuring client progress in a meaningful way and a line of sight to BEGIN to be able to connect interventions with client change and with successful labour market outcomes 85 What Was Learned?

 There remains much to do but there are strong indications that the approach can move us from a “what does not work” evaluation culture to a “what works”—a real transformation  A transformed evaluation culture must be predicated on buy-in from all levels and full recognition of complexity of evaluation and of client change; this is NOT a simple process and it is far from linear! 86 What Was Learned?

 Reports from practitioners that using PRIME made them more reflective about their own practice  Despite having to enter data twice, practitioners “hung in” with the research. This MATTERS!! 87 What Was Most Surprising?

 What have you learned?  What most surprised you?  What do you think are the implications for practice?  What next steps do you see as important? Reflective Questions…. 88

 Go to:  Click on Projects; Current Projects; From Research to Practice Symposium; Symposium Program  One page summaries of 8 additional applied research studies  Diverse populations: grades 6-12; unemployed/underemployed university graduates; social assistance recipients; laid off workers; disadvantaged adults 89 Other Research Studies

Lynne Bezanson and Sareena Hopkins Thank you! 90