1 SARAH HUNTER RAND CORPORATION LAURA STEIGHNER AMERICAN INSTITUTES FOR RESEARCH NOVEMBER 16, 2009 National Evaluation of the Demonstration to Improve the Direct Service Workforce

2 Briefing Outline Overview of Demonstrations and Evaluation Implementation Analysis  Methods  Findings General Lessons Learned

3 Project Team Structure Principal Investigator John Engberg, Ph.D. RAND Principal Investigator John Engberg, Ph.D. RAND Project Advisor Donna Farley, Ph.D. RAND Project Advisor Donna Farley, Ph.D. RAND Team Leaders Outcomes Evaluation John Engberg, Ph.D. RAND Nick Castle, Ph.D. Univ of Pittsburgh Outcomes Evaluation John Engberg, Ph.D. RAND Nick Castle, Ph.D. Univ of Pittsburgh Implementation Evaluation Sarah Hunter, Ph.D. RAND Laura Steighner, Ph.D. AIR Implementation Evaluation Sarah Hunter, Ph.D. RAND Laura Steighner, Ph.D. AIR

4 DEMONSTRATIONS EVALUATION PLAN MULTIPLE CASE STUDY APPROACH Overview of Study

5 Demonstrations Demonstrations were intended to improve recruitment and retention of direct service workers 5 grants awarded in 2003, 5 grants awarded in 2004 Grantees implemented various methods for accomplishing goals (details on next slide) Web-based reporting overseen by Lewin Technical assistance provided by Lewin and PHI Each site had a local evaluator

6 National Evaluation Purpose  To provide a summative evaluation that would allow a comparison across the sites Data Collection  Site visits  Surveys Evaluation  Process (implementation)  Outputs (participation, initiative cost)  Outcomes (job satisfaction, consumer satisfaction, turnover, retention, vacancies, turnover cost)

7 Multiple Case Study Approach Understand implementation context across different grantees  Describe initiatives, implementation, and their perceived outcomes  Identify patterns across grantees  Explore costs and sustainability Collected from multiple sources to increase validity  Reviewed extant documents (grant proposals, Lewin & PHI reports, quarterly reports)  Developed logic models in collaboration with grantees  Conducted telephone interviews and site visits and attended NFI conference

8 CROSSWALK OF INITIATIVES INITIATIVE Purpose Implementation Perceived Outcomes Implementation Analysis

9 Crosswalk of Initiatives InitiativesARDEINKYMENCNMVAVOAWA Health Care Coverage DSW Training Supervisor and Consumer Training Realistic Job Preview Peer Mentorship Merit-based or Longevity Recognition Worker Registry Marketing Campaign Targeted Recruitment Strategy

10 Health Care Purpose  To provide coverage or improve accessibility to health care Implementation  Demonstration is a difficult vehicle to study coverage  Variation across Grantees  Affordability  Sustainability  Skepticism over Free Good  Full-time status is a barrier to getting coverage  Health care coverage is complicated Perceived Outcomes  Need for Coverage  Past Unmet Need  Improved Climate

11 DSW Training Purpose  To improve competence and job satisfaction among DSWs Implementation  Training Content  Professional Development  Training Mode  Scheduling  Incentives for Participation and Completion  Participation by Supervisors/Managers  Voluntary versus Mandatory Participation Perceived Outcomes  Training Effectiveness  Community Building among DSWs  Improved DSW Self-Care

12 Supervisor / Consumer Supervisor Training Purpose  To improve DSW job satisfaction by improving communications with supervisors and/or consumers Implementation  Target Audience  Needs Assessment  Training Technique Perceived Outcomes  Training Effectiveness  Retention and Turnover  Community Building among Participants

13 Realistic Job Previews Purpose  To set realistic candidate expectations for the job and improve retention Implementation  Match between RJP Content and Job Context  Third-Party Implementation of RJP  Consumer Participation Perceived Outcomes  Person-Job Fit  Turnover  Quality of Care

14 Peer Mentoring Initiative Purpose  To provide support to newly hired workforce Implementation  Targeted new staff and mentors selected from veteran staff  Matching Process, Contact, Intensity, and Variability in Documentation across Grantees Perceived Outcomes  Resistance among Supervisory Staff  Mentors enjoyed program  Unlikely to Improve Retention

15 Recognition Purpose  To improve DSW retention by providing either merit-based and longevity incentives Implementation  Initiated longevity award with early retention goals (6 th months, 1 year) and later added longer retention awards to acknowledge all staff  Logging Tenure, Applying for Award  Type of Recognition  Public Recognition Perceived Outcomes  Staff Appreciation

16 Worker Registry Purpose  To improve both recruitment and retention by establishing better match between job and worker Implementation  Development Time  Worker Eligibility  Consumer Eligibility Perceived Outcomes  Income Stabilization  Balancing Supply and Demand  Consumer Choice  Employment and Costs

17 Marketing Purpose  To increase public awareness and recruit new DSWs to the field Implementation  Understanding the Community  Mode of Marketing Perceived Outcomes  Effectiveness of Mass Marketing Efforts  Effectiveness o f Targeted Marketing Efforts

18 Targeted Recruitment Strategies Purpose  To identify non-traditional populations to recruit to the field Implementation  Sources for Recruitment  Limited Resources or Staff to Recruit Candidates  Overcoming the Barriers to Working as a DSW  Case Management Approach Perceived Outcomes  Participation and Employment

19 MARKETING AND PARTICIPATION IMPLEMENTATION EVALUATION OUTCOMES General Lessons Learned

20 Marketing and Participation of Initiatives Research the population you are trying to reach Develop and showcase program champions Support behavioral change through case management and modeling

21 Implementation of Initiatives Consider who is in charge Initiatives take time Don’t overwhelm workforce with options Carefully plan incentives

22 Local Evaluation of Demonstration Consider the heterogeneity among worker sample Engage local evaluators early and often Don’t set up to assess a “moving target”

23 Demonstration Outcomes Consider ways to improve job satisfaction Initiatives may have positive, unanticipated benefits  Improved Agency Support  Worker Community Building Consider a “package” of initiatives

24 Contact Information Link to Report Sarah Hunter RAND (310) ext 7244 Laura Steighner AIR (202) technical_reports/TR699 Additional Information