1. Emerging Practices in VR: Moving from Pilot Project to System Integration 6 th Annual Summit on VR Program Evaluation & Quality Assurance September 16, 2013 Chip Kenney, Deborah Greene

2. Session Objectives Share methods used by Southeast TACE to evaluate integration of emerging practices in VR systems Review/Update findings from 2012 evaluation of Southeast TACE pilot projects designed to introduce practices. Present protocols and tools used to evaluate impact and measure/track systems integration. TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 2

3. Overview Since late 2009, the Southeast TACE has been partnering with a number of state vocational rehabilitation (VR) agencies to integrate innovative practices within the VR service delivery mainstream. Various states are piloting or integrating approaches such as Customized Employment and Asset Development. TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 3

4. Overview, cont. Five of these “Innovations Projects” have progressed to a point where their impact can be assessed to inform future action at local, state and regional levels. In 2012 TACE worked with project teams to evaluate individual project experiences as well as the Innovations initiative as a whole, in terms of their impact on the adoption of best practices within VR systems in the Southeast. TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 4

5. Overview, cont. All 5 projects are now taking steps to integrate piloted practices into VR delivery systems Three additional Innovations Projects are under way Evaluation of project activities continues through networking, tracking tools and Integration Scale measures TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 5

6. Overview, cont. Current status 5 Southeast states, 5 state agencies 8 projects in various stages of design or implementation, focused on…  Aspects of Customized Employment (7)  Asset Development (1) 5 have completed a pilot & are implementing further integration strategies 2 projects from original count on hold due to agency internal changes TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 6

7. Overview, cont. Summary of evaluation findings VRCs in 4 states have viable option to serve jobseekers with most significant disabilities, where previously they had no effective option Results directly attributable to pilot projects Value of investment evidenced in…  States’ further strategies to fully adopt the practice  Refinements/Evolution of the practice itself  Continuous improvement of the Innovations initiative New design/evaluation tools (Roadmap, Integration Scale, Community of Practice) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 7

8. Innovations strategy: Goal Increase the capacity of state agencies and their community partners to improve employment and self-sufficiency outcomes for individuals with significant and most disabilities TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 8

9. Innovations strategy: Premise Adoption of innovative practices requires more than training, hence a phased process from pilot to sustainability to integration Incubator to understand/address feasibility and systems implications before taking it to scale Partners can try the practice out, get their feet wet, make it their own, get buy-in Admin. and service delivery partners experience and influence the process – credibility, support for the practice and its feasibility Phases designed according to local needs, resources, readiness, logistics TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 9

10. Approach to Innovation Projects Roadmap TA/CE Pilot Impact Feasibility ROI Outcomes Evaluate Expansion Community of Practice Ongoing TA Sustain Integration Scale Integrate TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2012 10

11. Innovations strategy: Process Meet with agency leadership/staff re. goals, success measures, strategy Agree on multi-phase design Assign project manager/team Develop project roadmap (workplan, resources) Identify pilot sites, partners, resources Conduct performance-based training Facilitate partnerships and infrastructure supports Evaluate pilot phase Refine plans for expansion/sustainability/integration Launch subsequent phases TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 11

12. Description of Pilots Original 5 projects Florida (2)  Customized Employment - Transition  Discovery Certification - Transition and Adult Mississippi  Customized Employment - Adult Georgia  Customized Employment - Transition and Adult Kentucky  Customized Employment - Adult TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 12

13. Description of Pilots, cont. 3 new projects since 2012 Kentucky  Asset Development Georgia  Discovery in Residential Setting - Adult North Carolina  Customized Employment - Transition TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 13

14. Evaluation, cont. Design principles Demonstrate value of Innovations approach to support adoption of emerging practices Assess impact to date of individual projects and initiative as a whole Directly connect results with the intervention Establish process/protocols for ongoing evaluation and continuous improvement TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 14

15. Evaluation, cont. Targets Outputs  Trainings conducted, numbers trained  Meetings facilitated  Products generated (reports, tools, samples) Impact  Change in attitude/behavior, i.e., awareness/ understanding/utilization of the practice  Systems changes, e.g., partnerships and infrastructure measures to operationalize the practice TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 15

16. Evaluation, cont. Targets, cont. Return on investment: Start-up and operational costs, value-added Customer satisfaction: Agency/Partner assessment of project experience and quality of TACE services Jobseeker outcomes  Completed profiles  Job placements  Financial security measures TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 16

17. Evaluation, cont. Subjects: Projects that…  Completed the pilot or training phase (at least), and  Took or are considering measures to ramp up Types of data  Quantitative (trainings conducted, # participants, active practitioners, etc.)  Qualitative (partnerships & infrastructure, attitudes & behavior) Overall Innovations initiative as a TACE technical assistance activity TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 17

18. Evaluation, cont. Data sources Tracking tools, reports and other documentation (communications, products, etc.) Southeast TACE team (staff & consultants involved in development of initiative, design and execution of projects, etc.) Agency leadership Project leads & participants (incl. jobseekers and their families) Communities of practice TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 18

19. Evaluation, cont. Process Agree on measures, design evaluation strategy with input from TACE team, agencies, projects Develop/Update evaluation Work Plan Design data collection and interview protocol (source documents, key informant interviews, focus group) Collect/Analyze prelim. Data Draft individual project reports and aggregate discussion paper TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 19

20. Evaluation, cont. Process, cont. Conduct focus group among project leads Synthesize findings – focus group report Disseminate white paper summarizing findings and recommendations (TACE website) Publish article in Journal of Rehabilitation Administration Construct new evaluation tools  Project Roadmap  Integration Scale TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 20

21. Evaluation, cont. Process, cont. Convene communities of practice  Customized Employment  Asset Development Update evaluation measures and strategy  Outputs (projects, trainings, TA activities, certified or authorized practitioners)  Systems integration/impact indicators (policies, MOUs, curricula, expansion activities)  Operations indicators – changes in attitudes, practices  Client impact indicators – quantitative, qualitative TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 21

22. Evaluation, cont. Findings: Infrastructure CE/Discovery defined/endorsed as discrete services in policy/regulation (MS, GA, FL, KY) CE/Discovery incl. in VR strategic plans (FL, GA) Fee code changes (MS, GA, FL, KY) Long term supports  FL: DD agencies  GA: MOUs between GVRA and DD/MH  KY: HCBS waiver – require referral to VR  MS: HCBS waiver – in discussion TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 22

23. Evaluation, cont. Findings: Capacity-building Internal Capacity-building  Creation/Modification of positions to support CE  Creating mentor/support teams Provider/Partner Capacity-building  Outreach/Funding strategies to train/develop a pool of VR-approved CRPs (GA)  Communities of practice (FL, MS, GA)  Mentoring strategies (MS, FL, KY)  University-based training or credentialing program in progress (GA, FL, KY) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 23

24. Evaluation, cont. Findings: Partnerships School Districts (early collaboration for transition) Employment First DD and MH agencies Universities: capacity-building, evaluation Joint outreach & tracking strategies Shared recognition of practices as cross-system continuum of services requiring collaboration Reduced ‘finger pointing’ TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 24

25. Evaluation, cont. Findings: Systems Change Awareness of CE as an option and increasingly requesting CE or Discovery for a given jobseeker Understanding of the innovation as a change in VR practice Use of CE by VR systems practitioners TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 25

26. Roadmap Step-by-step guidance in developing Innovation projects Ensures successful implementation, fidelity  Objectives, success measures, evaluation  Phases  Stakeholders, partners  Expectations  Advance work  Workplan, resources  Deliverables TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 26

27. Roadmap, cont. Tools to plan project, conduct meetings, negotiate agreements, e.g.  SWOT analysis  Resource assessment  Readiness/Feasibility assessment  Risk assessment  Meeting agenda template Use for design, development, tracking and evaluation TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 27

28. Integration Scale Key indicators Agency Buy-in Policy and Procedures Capacity/Resources Data Collection Accountability/ Results Reporting Evaluation/Impact/ Continuous Improvement TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2012 28

29. Integration Scale: Agency Buy-In Agency Leadership Agency Champion Administrative Level Field Service Level Partner Level Success Measures incorporated in strategic plan, agency identity, accountability TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2012 29

30. Integration Scale: Policy/Procedures Policy documents, communications Fee structure that adequately compensates providers Field Service Manual Built into staff job descriptions, performance expectations (management, supervision, field services and support) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2012 30

31. Integration Scale: Capacity/Resources Competencies Identified Qualified Providers Skilled Staff (management, supervision, field services and support) Adequate financial resources Ongoing training Certification Process for Providers Mentoring Tools Technical Assistance Partners/ Formal Agreements TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2012 31

32. Integration Scale: Data Collection Tracking  Employment outcomes  Client satisfaction  Longitudinal Planned Service Purchased Service TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2012 32

33. Integration Scale: Accountability/ Results Reporting Performance expectations Reporting requirements Periodic reviews Community of Practice with designated agency liaison TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2012 33

34. Integration Scale: Eval./ Continuous Improvement Evaluation strategy, workplan Periodic review of results Results factored into strategic plan, milestones, infrastructure, capacity-building TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2012 34

35. Community of Practice Representatives of Innovation Projects meet at least quarterly  Customized Employment  Asset Development Ongoing technical assistance Peer networking, exchange Ongoing needs assessment Ongoing accountability and evaluation TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2012 35

36. Impact Measures Extent to which advances in integration can be attributed to TACE strategy Outputs (projects, trainings, TA activities, certified or authorized practitioners) Systems integration indicators (policies, MOUs, curricula, expansion activities) Operations indicators – changes in attitudes, knowledge; use of practices; referral rates Client impact indicators – quantitative, qualitative TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 36

37. Observations: Highlights Leadership willingness to take risks, devote resources to a practice untested in their system Systems changes that substantially increased capacity to serve jobseekers with the most significant disabilities Outcomes clearly attributable to the intervention Costs deemed worth the return on investment Pilot results are basis for decisions to expand, replicate, sustain the practice New products, new directions (e.g., residential) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 37

38. Observations: Challenges Buy-in: Reinforcing that this is about serving a population for whom VR traditionally has not had a successful strategy; that building capacity will take time/effort and will be a departure from business as usual Consistently engaging leadership & management commitment sufficient to support and sustain a project (even w/ personnel changes) Project planning to create environment for successful pilot TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 38

39. Observations: Challenges Articulating success measures directly connected to project activities Identifying terms/triggers for moving from one phase of implementation to another Prioritizing resources for project management and face-to-face training and technical assistance. Communicating project parameters, expectations Logistics: Scheduling training, assigning cases TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 39

40. Recommendations Phased approach for practices that are a significant departure from business as usual Success measures clearly connected to project activities and decisions to expand/ sustain the practice Project workplan covering all phases (pilot to evaluation/replication to full implementation) Project management and facilitation resources TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 40

41. Recommendations Clear communication of expectations/ implications in advance of committing to pilot Supplemental resources to finance project activities and ongoing operations Distance learning platforms rather than conference calls Leadership program for administrators providing info. on promising practices Supports to sustain/integrate practices (Roadmap, Integration Scale, Community of Practice) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 41

42. Questions & Discussion TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 42

43. TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 43 Southeast TACE Region IV Toll-free: (866) 518-7750 [voice/tty] Fax: (404) 541-9002 Web: TACEsoutheast.org MyTACE Portal: TACEsoutheast.org/myportal Email: tacesoutheast@law.syr.edu

44. TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A080021. © 2013 44 Disclaimer This presentation was developed by the TACE Center: Region IV ©2013 with funds from the U.S. Department of Education, Rehabilitation Services Administration (RSA) under the priority of Technical Assistance and Continuing Education Projects (TACE) – Grant #H264A080021. However, the contents of this presentation do not necessarily represent the policy of the RSA and you should not assume endorsement by the Federal Government [34 CFR 75.620 (b)].