Transforming the Culture of Evaluation in Career and Employment Services Saskatchewan Abilities Council Saskatchewan Education, Employment and Labour New Brunswick Post-Secondary Training, Education and Labour Canadian Career Development Foundation (CCDF) Goss-Gilroy Inc. Canadian Research Working Group on Evidence Based Practice in Career Development (CRWG)

 CCDF: Lynne Bezanson & Céline Renald  Dave Redekopp, Life-Role Development Group  Guylaine Michaud, Université de Sherbrooke  Ken Organ, Goss-Gilroy  Provincial Partners ◦ Managers and Practitioners ◦ Clients: all who agree to participate; no criteria except not one-off’s

Input  Process  Outcome Indicators of client change 1.Learning outcomes Knowledge and skills linked to intervention 2.Personal attribute outcomes Changes in self- esteem, self efficacy, well-being, self confidence, self- awareness) 3.Impact outcomes Impact of #1 & #2 on client’s life, e.g., employment status, training status, quality, fit, standard of livinga Activities to promote positive change Generic interventions Working alliance, client engagement Specific interventions and progress in: Job Readiness Career Decision Making Skill Enhancement Work Search Work Maintenance Life Circumstances The support you provide and progress in personal attributes: self-esteem, self efficacy, well-being, self confidence, self-awareness) Resources available Staff: Number of staff, level of training, type of training Funding: Budget Agency mandate Community resources Access to Employment Opportunities and Training Client Employment History Client Life Circumstances

 What common indicators are applicable across different client contexts, different client groups, different agencies, and different interventions?  What statements can be made about service effectiveness by tracking common indicators of inputs, processes and outcomes? If numbers permit, it will be possible to aggregate data to begin to address the ultimate question, “What kinds of interventions in what contexts produce what kinds of outcomes?”

◦ LABOUR MARKET REALITIES: Catchment Area Profile  Local labour market opportunities – what is the quality and range of work opportunities?  Educational opportunities – easily accessible or not?  Community supports/resources – adequate? accessible? ◦ CLIENT REALITIES: Client Employment Potential (demographic info)  Work experience  Level of education  Language  Health factors  Other 5

◦ PRACTITIONER PROFILE  Age  Gender  Experience  Education/Training ◦ SERVICES OFFERED BY YOUR OFFICE  Provided by managers 6

Process  Build & Maintain Working Alliance with Client  Conduct Employability Needs Assessment ◦ 5 Employability Dimensions and Additional Life Circumstances  Develop and Update Action Plan  Interventions (throughout 6 weeks)  Track Client Progress (as appropriate throughout 6 weeks)

 PRE-EMPLOYABILITY/JOB READINESS: Has the intention to seek employment and/or enter training but faces challenges, vulnerabilities and/or skill deficits that need to be addressed before the client can be successful in achieving employment/training goals  CAREER DECISION MAKING: Not yet clear about work/learning goals; needs to understand personal skills, interest, values and personality and more about different potential occupations in order to develop a vision for a preferred future  SKILLS ENHANCEMENT: Has career goal, but needs to explore skills and qualifications needed for the work they want and how to acquire them  WORK SEARCH: Has career goal and necessary skills/qualifications for desired work; needs skills, strategies and tools necessary to conduct a successful job search  EMPLOYMENT MAINTENANCE: Has difficulty keeping work or may need to work on the skills/attitudes required for success at work  OTHER: Many circumstances beyond those covered by the employability dimensions may impact on the client’s employability outcomes. Some of these life circumstances may necessitate referral outside of career and employment services. Life circumstance issues may need to be addressed and resolved before progress toward employment can realistically be made.

Client Progress Indicators  Employability Needs ◦ 5 Employability Dimensions ◦ Additional Life Circumstances  Personal Attributes Needs/Working Alliance ◦ Self-management, self-esteem, well-being, self-efficacy, self-awareness ◦ Working alliance ◦ Client engagement  Labour Market Indicators ◦ Tailored to client status

Not at all Not much A little Quite a lot A lot  Identify and clarify future direction (e.g.,training, education, employment or change in life circumstances goal)  Identify personal strengths/resources that support future direction (e.g.,training, education, employment or change in life circumstances goal)  Resolve specific challenges/vulnerabilities that may impact on future direction (e.g., mortgage, public transit, day care etc.)  Access community resources/supports that address specific personal challenges/vulnerabilities and assist in helping move goal forward (e.g., mental health services; addictions counselling; public housing etc.)  Develop necessary personal supports needed to move toward future direction (e.g., family, childcare, transportation)  Acquire life/employment skills and attitudes that support future direction  Acquire attitudes which support future direction  Develop and follow a plan of action to move future direction forward  Keep appointments and sustain efforts  Other: (please specify)________________________________

Indicators of challenges in client life circumstances are in evidence: Not at all Not much A little Quite a lot A lot  Improved housing is needed  Improved transportation is needed  Improved capacity to work and/or study is needed  Increase in sense of responsibility for own choices and behaviours is needed  Increase in ability to set short and long term goals is needed  Increase in understanding expectations and demands of employers is needed  Reduction in destructive behavior is needed  Following medical and medication protocols is needed  Increased openness to change is needed  Increased access to constructive and positive support systems is needed  Improved relationships with family and friends are needed  Increased trust in other people is needed  Other: _________________________________

Client needs assistance in the following areas: Not at all Not much A little Quite a lot A lot Ability to Self-Manage:  More info Developing stronger Self-Esteem:  More info Improving sense of Well-Being: i.e.:  More info Developing stronger Self-Efficacy: i.e.:  More info Increasing Self-Awareness:  More info  Other (Please specify): ________________________________________

 more info: Self-Esteem: “a person’s overall emotional evaluation of his or her own worth, an attitude to self and a judgement of oneself. Self-esteem encompasses beliefs (e.g., I am competent; I am worthy) and emotions (pride, shame). Self-esteem is the positive or negative evaluation of the self”  Demonstrates an optimistic outlook  Sees self as competent and able  Has confidence in ability to interact with others  Uses positive self-talk  Acts with little hesitation  Raises few objections  Asks questions  Articulates skills assertively  Makes eye contact  Is hopeful  Takes reasonable risks

Outcome Indicators  Employability Needs ◦ 5 Employability Dimensions ◦ Additional Life Circumstances  Personal Attributes Needs/Working Alliance ◦ Self-management, self-esteem, well-being, self-efficacy, self-awareness ◦ Working alliance ◦ Client engagement  Labour Market Indicators ◦ Tailored to client status  quality and “fit” of employment or education/training, salary, location

 Input & Outcome Connections ◦ More educated clients do better than less educated clients ◦ Success is positively related to labour market attachment ◦ “No Barrier” clients do better than “Single Barrier” clients who do better than “Multiple Barrier” clients ◦ Success is positively related to entry self-efficacy ◦ Success is positively related to entry self-esteem

 Outcomes and Processes ◦ Learning Outcomes  Client skill/knowledge increases with any intervention ◦ Personal Attribute Outcomes  Client attributes improve with any intervention  Client self-esteem improves with any intervention  Client self-efficacy improves with any intervention ◦ Labour Market Outcomes  Employment/training increases with any intervention  Quality/fit increases with any intervention  Working alliance is positively related to employment/ training outcomes

 Relationships between Outcomes ◦ Learning outcomes are positively related to:  personal attribute outcomes  labour market outcomes ◦ Personal attribute outcomes are positively related to labour market outcomes  Personal Attribute Outcomes & Processes ◦ Working alliance is positively related to self-efficacy changes ◦ Working alliance is positively related to self-esteem changes

 Learning Outcomes & Processes ◦ Client skill/knowledge increases are positively correlated with working alliance  Measurement Relationships ◦ Practitioner and client assessments of working alliance are positively related ◦ Practitioner assessment of personal attributes is positively related to client self-esteem scores ◦ Practitioner assessment of personal attributes is positively related to client self-efficacy scores

Input  Process  Outcome Indicators of client change 1.Learning outcomes Knowledge and skills linked to intervention 2.Personal attribute outcomes Changes in self- esteem, self efficacy, well-being, self confidence, self- awareness) 3.Impact outcomes Impact of #1 & #2 on client’s life, e.g., employment status, training status, quality, fit, standard of livinga Activities to promote positive change Generic interventions Working alliance, client engagement Specific interventions and progress in: Job Readiness Career Decision Making Skill Enhancement Work Search Work Maintenance Life Circumstances The support you provide and progress in personal attributes: self-esteem, self efficacy, well-being, self confidence, self-awareness) Resources available Staff: Number of staff, level of training, type of training Funding: Budget Agency mandate Community resources Access to Employment Opportunities and Training Client Employment History Client Life Circumstances

 More evidence for the field to justify the power and efficacy of career interventions  More integrity around outcomes  A culture of evaluation built into career and employment service practice  Over time, the ability to answer: ◦ “What kinds of interventions in what contexts produce what kinds of outcomes” 20

Research Reports will be completed by May 31, 2013 Follow the CRWG website at: Thank You