Emerging Practices in VR: Moving from Pilot Project to System Integration 6 th Annual Summit on VR Program Evaluation & Quality Assurance September 16,

Slides:



Advertisements
Similar presentations
Manatt manatt | phelps | phillips New York State Health Information Technology Summit Initiative Overview and Update Rachel Block, Project Director United.
Advertisements

LAO PDR Summary Findings from NOSPA Mission and Possible Next Steps.
Opening Doors: Federal Strategic Plan to Prevent and End Homelessness
Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.
State Plan for Independent Living UPDATE Overview, Impact and Involvement.
August 2006 OSEP Project Director's Conference 1 Preparing Teachers to Teach All Children: The Impact of the Work of the Center for Improving Teacher Quality.
Sustainability Planning Pat Simmons Missouri Department of Health and Senior Services.
Evaluation of the Impact of Pilot Projects on State VR Agencies’ Delivery Systems 5 th Annual Summit VR Program Evaluation & Quality Assurance September.
Maine SIM Evaluation: Presentation to Steering Committee December 10, 2014.
Regional Performance and Innovative Consortia(RPIC) Georgia Community Action Association Annual Conference Author: Lorraine Daniels, Executive Director.
Family Resource Center Association January 2015 Quarterly Meeting.
Presented at Annual Conference of the American Evaluation Association Anaheim, CA, November 2011 Lessons Learned about How to Support Outcomes Measurement.
Action Implementation and Monitoring A risk in PHN practice is that so much attention can be devoted to development of objectives and planning to address.
Quality evaluation and improvement for Internal Audit
PHAB's Approach to Internal and External Evaluation Jessica Kronstadt | Director of Research and Evaluation | November 18, 2014 APHA 2014 Annual Meeting.
Behavioral Economics Asset Development & Vocational Rehabilitation 30-Second Training Series.
IS&T Project Management: How to Engage the Customer September 27, 2005.
Family Outreach and Response Program Strategic Plan September 26 th, 2012.
Asset Development: Student Earned Income Exclusion Asset Development & Vocational Rehabilitation 30-Second Training Series.
Bureau of Rehabilitation Services (BRS) Department of Rehabilitation Services (DORS) Understanding the State Plan for Vocational Rehabilitation and Supported.
1 EEC Board Policy and Research Committee October 2, 2013 State Advisory Council (SAC) Sustainability for Early Childhood Systems Building.
Experiences with Expanded Discovery and Internships for Transition Aged Youth Therese Fimian Institute for Economic Empowerment Norciva Shumpert Marc Gold.
The Role of Work Experiences in Guiding Careers Richard Luecking, Ed.D. TransCen, Inc. October 27, 2010.
Position Your Employment Services in the Labor Market Define your Employment Services Describe the Feature and Benefits of your Employment Services Develop.
A Charge to Collaborate: IT’S NOT JUST ABOUT WHAT WE DO… IT’S ABOUT HOW WE DO IT…
Alaska’s Experience.  Lack of policy that promotes work as an expectation (or employment first)  Fear of losing health benefits  Financial disincentives.
Every Woman Southeast REPRODUCTIVE LIFE PLANNING COMMUNITY GRANTS
WHAT IS “CLASS”? A BRIEF ORIENTATION TO THE CLASS METHODOLOGY.
The Vision Implementation Project
Focus on Employment Provider Council Meeting November 17, 2006 Beth McArthur.
The Role of State VR Agencies in Workforce Investment Systems: Creating a Universally Accessible and Seamless System that Meets the Needs of Persons with.
Creating a New Vision for Kentucky’s Youth Kentucky Youth Policy Assessment How can we Improve Services for Kentucky’s Youth? September 2005.
DELAWARE HEALTH AND SOCIAL SERVICES Division of State Service Centers Office of Volunteerism The Corporation for National and Community Service.
The RRCP Program A Framework for Change Presented to our SPDG Partners June 2010.
Do it pro bono. Strategic Scorecard Service Grant The Strategy Management Practice is presented by Wells Fargo. The design of the Strategic Scorecard Service.
Vocational Rehabilitation And The New Ticket to Work Program Presenters Harold Thornton, TACE Consultant Elizabeth Jennings, National Disability Institute.
The Arizona Community of Practice (AzCoPT) June 12,
Engagement as Strategy: Leading by Convening in the SSIP Part 2 8 th Annual Capacity Building Institute May, 2014 Joanne Cashman, IDEA Partnership Mariola.
Asset Development: Bank On Asset Development & Vocational Rehabilitation 30-Second Training Series.
Asset Development: Substantial Gainful Activity (SGA) Asset Development & Vocational Rehabilitation 30-Second Training Series.
Fundamentals of Evaluation for Public Health Programs ROBERT FOLEY, M.ED. NIHB TRIBAL PUBLIC HEALTH SUMMIT MARCH 31,
Working Definition of Program Evaluation
Clean, Green, and on the Same Team! How Economic and Workforce Development are Partnering to Serve an Emerging Regional Industry.
1. 2 Collaborative Partnerships It’s that evolution thing again! Adult education has been partnering and collaborating for years.
California Statewide Prevention and Early Intervention (PEI) Projects Overview May 20, 2010.
United We Ride: Where are we Going? December 11, 2013 Rik Opstelten United We Ride Program Analyst.
12/07/20101 Bidder’s Conference Call: ARRA Early On ® Electronic Enhancement to Individualized Family Service Plans (EE-IFSP) Grant and Climb to the Top.
VR Counselors Working with Schools During Transition Laura Spears & Kelley Ali Transition Specialists, South Carolina Vocational Rehabilitation Department.
Service Collaboration & Linkages to Achieve Seamless Transition from School to Career Outcomes Debra Martin Luecking Senior Executive for LueMar Consulting.
HECSE Quality Indicators for Leadership Preparation.
Online Toolkit for Job Placement and Employment Professionals Dennis Gilbride, Ph.D., C.R.C., Georgia State University TACE Consultant December 4, 2012.
The Four Steps to an Employer Relationship Bill Santos President/Partner of Employment Management Professionals July 15, 2014.
Promoting school connections for youth in child welfare Joan Rock, Implementation Coordinator Kristen Hayden-West, Implementation Coordinator Jessica Strolin-Goltzman,
Implementing QI Projects Title I HIV Quality Management Program Case Management Providers Meeting May 26, 2005 Presented by Lynda A. O’Hanlon Title I HIV.
TACE Developing a Community of Excellence Marc Gold & Associates December 18, 2008.
Order of Selection: Lessons Learned From a Study of Federal and Selected State Policy Frameworks Presented by: Robert “Bobby” Silverstein December 12,
The Way to Work: Employment Preparation for Youth in Transition Richard Luecking, Ed.D. TransCen, Inc January 9, 2014.
Office of Special Education Programs U.S. Department of Education GRANT PERFORMANCE REPORT FOR CONTINUATION FUNDING.
CSEFEL State Planning Rob Corso. CSEFEL  National Center focused on promoting the social emotional development and school readiness of young children.
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
UNC Deans Council The North Carolina K-12 Digital Learning Transition Glenn Kleiman Friday Institute for Educational Innovation NC State University College.
1 Community-Based Care Readiness Assessment and Peer Review Overview Department of Children and Families And Florida Mental Health Institute.
Friday Institute Leadership Team Glenn Kleiman, Executive Director Jeni Corn, Director of Evaluation Programs Phil Emer, Director of Technology Planning.
PERKINS IV AND THE WORKFORCE INNOVATION AND OPPORTUNITY ACT (WIOA): INTERSECTIONS AND OPPORTUNITIES.
1 A Multi Level Approach to Implementation of the National CLAS Standards: Theme 1 Governance, Leadership & Workforce P. Qasimah Boston, Dr.Ph Florida.
2016 Spring Grantee Convening IKF Evaluation Update Center for Community Health and Evaluation April 11, 2016 Foundation for a Healthy Kentucky.
Minnesota’s Disability Employment Initiative (DEI): Partners for Youth Third Round of DEI Grants October 2013.
HRSA Early Childhood Comprehensive Systems (ECCS) Impact 2016 Funding Opportunity Announcement (FOA) Barbara Hamilton, Project Officer Division.
Presented at Annual Conference of the American Evaluation Association Anaheim, CA, November 2011 Lessons Learned about How to Support Outcomes Measurement.
2018 OSEP Project Directors’ Conference
Presentation transcript:

Emerging Practices in VR: Moving from Pilot Project to System Integration 6 th Annual Summit on VR Program Evaluation & Quality Assurance September 16, 2013 Chip Kenney, Deborah Greene

Session Objectives Share methods used by Southeast TACE to evaluate integration of emerging practices in VR systems Review/Update findings from 2012 evaluation of Southeast TACE pilot projects designed to introduce practices. Present protocols and tools used to evaluate impact and measure/track systems integration. TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Overview Since late 2009, the Southeast TACE has been partnering with a number of state vocational rehabilitation (VR) agencies to integrate innovative practices within the VR service delivery mainstream. Various states are piloting or integrating approaches such as Customized Employment and Asset Development. TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Overview, cont. Five of these “Innovations Projects” have progressed to a point where their impact can be assessed to inform future action at local, state and regional levels. In 2012 TACE worked with project teams to evaluate individual project experiences as well as the Innovations initiative as a whole, in terms of their impact on the adoption of best practices within VR systems in the Southeast. TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Overview, cont. All 5 projects are now taking steps to integrate piloted practices into VR delivery systems Three additional Innovations Projects are under way Evaluation of project activities continues through networking, tracking tools and Integration Scale measures TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Overview, cont. Current status 5 Southeast states, 5 state agencies 8 projects in various stages of design or implementation, focused on…  Aspects of Customized Employment (7)  Asset Development (1) 5 have completed a pilot & are implementing further integration strategies 2 projects from original count on hold due to agency internal changes TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Overview, cont. Summary of evaluation findings VRCs in 4 states have viable option to serve jobseekers with most significant disabilities, where previously they had no effective option Results directly attributable to pilot projects Value of investment evidenced in…  States’ further strategies to fully adopt the practice  Refinements/Evolution of the practice itself  Continuous improvement of the Innovations initiative New design/evaluation tools (Roadmap, Integration Scale, Community of Practice) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Innovations strategy: Goal Increase the capacity of state agencies and their community partners to improve employment and self-sufficiency outcomes for individuals with significant and most disabilities TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Innovations strategy: Premise Adoption of innovative practices requires more than training, hence a phased process from pilot to sustainability to integration Incubator to understand/address feasibility and systems implications before taking it to scale Partners can try the practice out, get their feet wet, make it their own, get buy-in Admin. and service delivery partners experience and influence the process – credibility, support for the practice and its feasibility Phases designed according to local needs, resources, readiness, logistics TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Approach to Innovation Projects Roadmap TA/CE Pilot Impact Feasibility ROI Outcomes Evaluate Expansion Community of Practice Ongoing TA Sustain Integration Scale Integrate TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Innovations strategy: Process Meet with agency leadership/staff re. goals, success measures, strategy Agree on multi-phase design Assign project manager/team Develop project roadmap (workplan, resources) Identify pilot sites, partners, resources Conduct performance-based training Facilitate partnerships and infrastructure supports Evaluate pilot phase Refine plans for expansion/sustainability/integration Launch subsequent phases TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Description of Pilots Original 5 projects Florida (2)  Customized Employment - Transition  Discovery Certification - Transition and Adult Mississippi  Customized Employment - Adult Georgia  Customized Employment - Transition and Adult Kentucky  Customized Employment - Adult TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Description of Pilots, cont. 3 new projects since 2012 Kentucky  Asset Development Georgia  Discovery in Residential Setting - Adult North Carolina  Customized Employment - Transition TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Design principles Demonstrate value of Innovations approach to support adoption of emerging practices Assess impact to date of individual projects and initiative as a whole Directly connect results with the intervention Establish process/protocols for ongoing evaluation and continuous improvement TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Targets Outputs  Trainings conducted, numbers trained  Meetings facilitated  Products generated (reports, tools, samples) Impact  Change in attitude/behavior, i.e., awareness/ understanding/utilization of the practice  Systems changes, e.g., partnerships and infrastructure measures to operationalize the practice TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Targets, cont. Return on investment: Start-up and operational costs, value-added Customer satisfaction: Agency/Partner assessment of project experience and quality of TACE services Jobseeker outcomes  Completed profiles  Job placements  Financial security measures TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Subjects: Projects that…  Completed the pilot or training phase (at least), and  Took or are considering measures to ramp up Types of data  Quantitative (trainings conducted, # participants, active practitioners, etc.)  Qualitative (partnerships & infrastructure, attitudes & behavior) Overall Innovations initiative as a TACE technical assistance activity TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Data sources Tracking tools, reports and other documentation (communications, products, etc.) Southeast TACE team (staff & consultants involved in development of initiative, design and execution of projects, etc.) Agency leadership Project leads & participants (incl. jobseekers and their families) Communities of practice TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Process Agree on measures, design evaluation strategy with input from TACE team, agencies, projects Develop/Update evaluation Work Plan Design data collection and interview protocol (source documents, key informant interviews, focus group) Collect/Analyze prelim. Data Draft individual project reports and aggregate discussion paper TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Process, cont. Conduct focus group among project leads Synthesize findings – focus group report Disseminate white paper summarizing findings and recommendations (TACE website) Publish article in Journal of Rehabilitation Administration Construct new evaluation tools  Project Roadmap  Integration Scale TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Process, cont. Convene communities of practice  Customized Employment  Asset Development Update evaluation measures and strategy  Outputs (projects, trainings, TA activities, certified or authorized practitioners)  Systems integration/impact indicators (policies, MOUs, curricula, expansion activities)  Operations indicators – changes in attitudes, practices  Client impact indicators – quantitative, qualitative TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Findings: Infrastructure CE/Discovery defined/endorsed as discrete services in policy/regulation (MS, GA, FL, KY) CE/Discovery incl. in VR strategic plans (FL, GA) Fee code changes (MS, GA, FL, KY) Long term supports  FL: DD agencies  GA: MOUs between GVRA and DD/MH  KY: HCBS waiver – require referral to VR  MS: HCBS waiver – in discussion TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Findings: Capacity-building Internal Capacity-building  Creation/Modification of positions to support CE  Creating mentor/support teams Provider/Partner Capacity-building  Outreach/Funding strategies to train/develop a pool of VR-approved CRPs (GA)  Communities of practice (FL, MS, GA)  Mentoring strategies (MS, FL, KY)  University-based training or credentialing program in progress (GA, FL, KY) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Findings: Partnerships School Districts (early collaboration for transition) Employment First DD and MH agencies Universities: capacity-building, evaluation Joint outreach & tracking strategies Shared recognition of practices as cross-system continuum of services requiring collaboration Reduced ‘finger pointing’ TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Evaluation, cont. Findings: Systems Change Awareness of CE as an option and increasingly requesting CE or Discovery for a given jobseeker Understanding of the innovation as a change in VR practice Use of CE by VR systems practitioners TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Roadmap Step-by-step guidance in developing Innovation projects Ensures successful implementation, fidelity  Objectives, success measures, evaluation  Phases  Stakeholders, partners  Expectations  Advance work  Workplan, resources  Deliverables TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Roadmap, cont. Tools to plan project, conduct meetings, negotiate agreements, e.g.  SWOT analysis  Resource assessment  Readiness/Feasibility assessment  Risk assessment  Meeting agenda template Use for design, development, tracking and evaluation TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Integration Scale Key indicators Agency Buy-in Policy and Procedures Capacity/Resources Data Collection Accountability/ Results Reporting Evaluation/Impact/ Continuous Improvement TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Integration Scale: Agency Buy-In Agency Leadership Agency Champion Administrative Level Field Service Level Partner Level Success Measures incorporated in strategic plan, agency identity, accountability TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Integration Scale: Policy/Procedures Policy documents, communications Fee structure that adequately compensates providers Field Service Manual Built into staff job descriptions, performance expectations (management, supervision, field services and support) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Integration Scale: Capacity/Resources Competencies Identified Qualified Providers Skilled Staff (management, supervision, field services and support) Adequate financial resources Ongoing training Certification Process for Providers Mentoring Tools Technical Assistance Partners/ Formal Agreements TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Integration Scale: Data Collection Tracking  Employment outcomes  Client satisfaction  Longitudinal Planned Service Purchased Service TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Integration Scale: Accountability/ Results Reporting Performance expectations Reporting requirements Periodic reviews Community of Practice with designated agency liaison TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Integration Scale: Eval./ Continuous Improvement Evaluation strategy, workplan Periodic review of results Results factored into strategic plan, milestones, infrastructure, capacity-building TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Community of Practice Representatives of Innovation Projects meet at least quarterly  Customized Employment  Asset Development Ongoing technical assistance Peer networking, exchange Ongoing needs assessment Ongoing accountability and evaluation TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Impact Measures Extent to which advances in integration can be attributed to TACE strategy Outputs (projects, trainings, TA activities, certified or authorized practitioners) Systems integration indicators (policies, MOUs, curricula, expansion activities) Operations indicators – changes in attitudes, knowledge; use of practices; referral rates Client impact indicators – quantitative, qualitative TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Observations: Highlights Leadership willingness to take risks, devote resources to a practice untested in their system Systems changes that substantially increased capacity to serve jobseekers with the most significant disabilities Outcomes clearly attributable to the intervention Costs deemed worth the return on investment Pilot results are basis for decisions to expand, replicate, sustain the practice New products, new directions (e.g., residential) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Observations: Challenges Buy-in: Reinforcing that this is about serving a population for whom VR traditionally has not had a successful strategy; that building capacity will take time/effort and will be a departure from business as usual Consistently engaging leadership & management commitment sufficient to support and sustain a project (even w/ personnel changes) Project planning to create environment for successful pilot TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Observations: Challenges Articulating success measures directly connected to project activities Identifying terms/triggers for moving from one phase of implementation to another Prioritizing resources for project management and face-to-face training and technical assistance. Communicating project parameters, expectations Logistics: Scheduling training, assigning cases TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Recommendations Phased approach for practices that are a significant departure from business as usual Success measures clearly connected to project activities and decisions to expand/ sustain the practice Project workplan covering all phases (pilot to evaluation/replication to full implementation) Project management and facilitation resources TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Recommendations Clear communication of expectations/ implications in advance of committing to pilot Supplemental resources to finance project activities and ongoing operations Distance learning platforms rather than conference calls Leadership program for administrators providing info. on promising practices Supports to sustain/integrate practices (Roadmap, Integration Scale, Community of Practice) TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

Questions & Discussion TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A ©

TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A © Southeast TACE Region IV Toll-free: (866) [voice/tty] Fax: (404) Web: TACEsoutheast.org MyTACE Portal: TACEsoutheast.org/myportal

TACE Center: Region IV, a project of the Burton Blatt Institute. Funded by RSA Grant # H264A © Disclaimer This presentation was developed by the TACE Center: Region IV ©2013 with funds from the U.S. Department of Education, Rehabilitation Services Administration (RSA) under the priority of Technical Assistance and Continuing Education Projects (TACE) – Grant #H264A However, the contents of this presentation do not necessarily represent the policy of the RSA and you should not assume endorsement by the Federal Government [34 CFR (b)].