EFLSMP Summative Evaluation Social Dynamics, LLC 481 North Frederick Ave., Suite 410 Gaithersburg, MD 20877 Phone: (301) 990-1105 http://www.socialdynamicsllc.com EFLSMP Summative Evaluation Douglas Klayman, Ph.D. September 2016

Overview Program Characteristics State Topics Program Components Summative Evaluation Methodology Data Sources Measuring Systems Change and Increases in Capacity General Observations and Conclusions Employment First is really a framework for systems change that is centered on the premise that all citizens, including individuals with significant disabilities, are capable of full participation in integrated employment and community life Integrated, Competitive Employment (ICE): Jobs that people want; are located in competitive, integrated settings; and that pay at least the federal minimum wage EFSLMP assists states in may changes in their systems and capacity to provide competitive integrated employment. One of the key approaches is rate restructuring along with provider transformation pilot projects.

Program Characteristics Concepts Federal Initiative Employment First: A framework for systems change that is centered on the premise that all citizens, including individuals with significant disabilities, are capable of full participation in integrated employment and community life Integrated, Competitive Employment (ICE): Jobs that people want; are located in competitive, integrated settings; and that pay at least the federal minimum wage Employment First State Leadership Mentoring Program (EFSLMP): The vehicle for implementing Systems Change, Capacity Building and states’ ability to Transform employment services from sheltered/day programs to ICE The various tiers of technical resources offered under EFSLMP have expanded over the years. Vision Question Three-tiered program: Tier 1 (SME-facilitated T/TA services) and Tier 2 (VQ strategic and organizational planning activities) are offered to all core states while Tier 3 (Community of Practice) is available to any interested stakeholder (advocate, direct support professional, provider, government official, and others) in all 50 states and the District of Columbia. The sample of nine core states that are included in the 2015 summative evaluation had direct access to all three EFSLMP resource tiers. In 2015, the CoP provided webinars on a range of policy and practice innovations, such as benchmarking, development and maintenance of governing boards, provider transformation and its impact on staff, customized employment from the employer’s perspective, engagement of employers in the recruitment and employment of individuals with significant disabilities, competitive integrated employment, IPS, staff competencies for customized employment and self-employment, decentralization of service delivery systems, pre-vocational services and WIOA provisions.  

Program Characteristics cont’d Federal Employment Systems Change and Capacity Building Initiative Employment First State Leadership Mentoring Program (EFSLMP): The vehicle for implementing Systems Change, Capacity Building and states’ ability to Transform employment services from sheltered/day Systems change related to the policy landscape in support of competitive integrated employment EFSLMP assists states in identifying and addressing changes in state policies, funding models, and practices to increase access to and outcomes associated with competitive integrated employment. Employment First is really a framework for systems change that is centered on the premise that all citizens, including individuals with significant disabilities, are capable of full participation in integrated employment and community life Integrated, Competitive Employment (ICE): Jobs that people want; are located in competitive, integrated settings; and that pay at least the federal minimum wage EFSLMP assists states in may changes in their systems and capacity to provide competitive integrated employment. One of the key approaches is rate restructuring along with provider transformation pilot projects.

State Topics Provider Transformation Director Support Prof Capacity Building in Effective Practices School-to-Work Transition Employer Engagement Strategies Innovative Payment Schemes & Rate Restructuring Leveraging & Coordination of Policy and Resources across Systems Rate restructuring continues to be a very important issue for many states. Combined with provider transformation are the two topics that were consistently part of the program. School to work transition, Direct Support effective practices, employer engagement and leveraging policy and resources.

Other Topics Additional topics covered: Cross-systems change Implementation of promising practices Individual Placement and Support Performance measurement Policy development and guidance Effective hiring practices for employers Employer engagement in the recruitment and employment of individuals with disabilities Development of executive teams for cross-systems collaboration WIOA policy implementation and support and development of HCBS waivers and modifications

Program Components Tier 1: Intensive Training and Technical Assistance Purpose: To provide states with hands-on T/TA that supports the implementation of EF principles Availability: Core States Scope: Narrow (topics selected by state team leaders) Frequency: Varies (2–4 time per year) Facilitator: SME Result: SME works with selected members of the state team, state, and/or provider staff to provide information (e.g., provider transformation, policy analysis, advocacy, skill building)   Tier 2: Vision Quest To engage states in developing strategies for resolving challenges related to EF implementation Moderate (topics selected by state team) Quarterly State team works with SME to resolve challenges Tier 3: Community of Practice To learn about promising practices and strategies for EF implementation All states Broad (topics selected by ODEP and SMEs) High (monthly) SME or guest presenter Promising practices can be adopted by EF states for implementation There are three level of T/TA.

Summative Evaluation Methodology Objective: Identify systems change and increases in capacity to provide competitive integrated employment opportunities for individuals with significant disabilities Qualitative Data Collection: Collection and coding of systems change and capacity building products. Sample: Purposive sampling methodology for heterogeneity used to select nine core states out of the universe of 15. Geographically representative of the universe of the 2015 EFSLMP core states and chosen.

Data Sources Baseline and follow-up interviews with state collaborative team leaders 60-90 minute stakeholder interviews Detailed reviews of core state Technical Assistance Plans (TAP) Assessments of systems change and capacity-building products Reports and recordings s from Vision Quest (VQ) sessions T/TA webinars Professional development trainings provided by subject matter experts Monthly core state reports, We used baseline and two follow-up interviews to collect detailed information. This list is really in order of the importance of the information we collected. The protocol was aligned with the technical assistance plan (TAP). Let me show you an example of the TAP. It was really the key document that facilitated our data collection approach.

Measuring Systems Change and Increases in Capacity cont’d ODEP’s Criteria for Performance Excellence in Employment First State Systems Change and Provider Transformation Three systems change and two capacity building factors: 1. Monitoring and reacting to changes in demand among individuals with significant disabilities for competitive, integrated employment services (systems change factor); 2. Use of evidence-based practices (capacity building factor); 3. Changes in legislation/policy that positively affects individuals with disabilities (systems change factor); 4. Improvements in cost-effectiveness of services (systems change factor); 5. Use of rigorous performance measurement procedures that include key metrics such as employment placement and retention outcomes and receipt of prevailing wages (capacity building factor). This slide shows the systems change framework. It was developed by ODEP and adopted by us as a way to determine if the product (or document) focused on systems change and/or capacity building. For example, a product that documents changes in demand among individuals with disabilities for competitive, integrated employment, we categorized as a systems change product. One that focuses on evidence based practices and the use of new practices, we categorized as a capacity building factor. Performance measurement procedures was also a capacity building factors. So, for example, if we are concerned with the first factor, which is a systems change factor, we directed our research on each product, including interviews with stakeholders, to give us information on how a given product, increased demand for integrated employment. If it was a capacity building product, we would ask questions of respondents related to its ability to increase capacity for competitive integrated employment. For every product we had, and there were many, they were categorized based on one or more of these factors.

Measuring Systems Change and Increases in Capacity cont’d Three systems change and two capacity building factors: 1. Systems Change Factor 1: Monitoring/Reacting to Changes in Demand 2. Capacity Building Factor 1: Evidence-Based Practices 3. Systems Change Factor 2: Changes in Legislation/Policy 4. Systems Change Factor 3: Improvements in Cost-Effectiveness of Services *5. Use of rigorous performance measurement procedures that include key metrics such as employment placement and retention outcomes and receipt of prevailing wages (capacity building factor). This slide shows the systems change framework. It was developed by ODEP and adopted by us as a way to determine if the product (or document) focused on systems change and/or capacity building. For example, a product that documents changes in demand among individuals with disabilities for competitive, integrated employment, we categorized as a systems change product. One that focuses on evidence based practices and the use of new practices, we categorized as a capacity building factor. Performance measurement procedures was also a capacity building factors. So, for example, if we are concerned with the first factor, which is a systems change factor, we directed our research on each product, including interviews with stakeholders, to give us information on how a given product, increased demand for integrated employment. If it was a capacity building product, we would ask questions of respondents related to its ability to increase capacity for competitive integrated employment. For every product we had, and there were many, they were categorized based on one or more of these factors. *Not used by any state in evaluation.

Measuring Systems Change and Increases in Capacity cont’d Examples Systems Change Factor 1: Monitoring/Reacting to Changes in Demand Systems Change Factor 2: Changes in Legislation/Policy Systems Change Factor 3: Improvements in Cost-Effectiveness of Services Capacity Building Factor 1: Evidence-Based Practices New Partnership Linking VR with Workforce New HCBS Waiver Rate Reimbursement/ Restructuring This slide shows the systems change framework. It was developed by ODEP and adopted by us as a way to determine if the product (or document) focused on systems change and/or capacity building. For example, a product that documents changes in demand among individuals with disabilities for competitive, integrated employment, we categorized as a systems change product. One that focuses on evidence based practices and the use of new practices, we categorized as a capacity building factor. Performance measurement procedures was also a capacity building factors. So, for example, if we are concerned with the first factor, which is a systems change factor, we directed our research on each product, including interviews with stakeholders, to give us information on how a given product, increased demand for integrated employment. If it was a capacity building product, we would ask questions of respondents related to its ability to increase capacity for competitive integrated employment. For every product we had, and there were many, they were categorized based on one or more of these factors. Provider Transformation

Measuring Systems Change and Increases in Capacity cont’d STEP 1: Connect “on the ground” systems change and capacity building to the broad areas defined by ODEP For example, using the following T/TA topics: “policy development and alignment” is associated with factors 1 & 3. “provider transformation designed to upgrade the skills and knowledge of provider staff/provider transformation” is associated with factor 2. “rate reimbursement & restructuring” is associated with factor 4 *Factor 5, use of rigorous performance measurement procedures, was included only in Delaware’s TAP. Delaware was not selected for the evaluation and therefore, we excluded factor 5 from the 2015 Summative Evaluation.

Measuring Systems Change and Increases in Capacity Factor 1 Factor 2 Factor 3 Factor 4 MATRIX Selected EFSLMP Product Topics “Policy development and alignment” “provider transformation designed to upgrade the skills and knowledge of provider staff/provider transformation” “rate reimbursement & restructuring” √ √ √ √

Measuring Systems Change and Increases in Capacity cont’d STEP 2: Create systems change, capacity building and TAP milestone rubrics (See Separate PDFs Handouts) Systems change and capacity building rubric has three positions: No change in capacity or systems Pending: change that may happen either subsequent to authorization by an executive member of a state agency or the successful completion of a capacity building project or event. Completed : Products that have been implemented and as a result, transformed policy and practice during the current fiscal year.

Measuring Systems Change and Increases in Capacity cont’d STEP 3: Increases in Capacity: Measured dichotomously due to the limitations of the data and its lack of clearly defined gradations in the capacity building variable. Capacity defined: A perceptible increase in the capacity of providers within a state to provide services to individuals with disabilities that lead to competitive integrated employment opportunities. There are no graduations in the way we defined increased capacity. Products created by each state were rated separately and could be categorized as both systems change and capacity building.

Measuring Systems Change and Increases in Capacity 10 Systems Change Products Systems change products included documents that created state agency collaborations and led to formal partnerships Memoranda of Understanding Interagency cooperative agreements Modifications to existing policies and practices Agency directives Sub-regulatory policy guidance State legislation Changes to service definitions and rate structures in VR HCBS waivers and modifications

Measuring Systems Change and Increases in Capacity 23 Capacity Building Products Capacity-building products included resources that provide technical information on EF policies and practices: Resource guides Tool kits Data collection instruments Training curricula Fact sheets State and local conferences T/TA and provider trainings

Measuring Systems Change and Increases in Capacity 10 Systems Change Products and 23 Capacity Building Products

General Observations and Conclusions Many states developed cross-system partnerships and collaborations connecting Medicaid, Rehabilitation, Intellectual and Developmental Disabilities, Workforce Development, Workforce, Mental Health, Education, and VR agencies with each other through their EFLSMP state collaborative teams. Organizational inertia caused by limitations in fiscal and/or human resources, slowed process of policy acceptance and implementation Changing the “status quo” and transforming state systems depended on how collaborative state teams used their capacity-building and systems change products to communicate with state leaders, service providers, and other stakeholder groups.

General Observations and Conclusions Most state collaborative teams expedited the policy process. Some were challenged by a an extended period of time waiting for final clearance or authorization from executive leadership. Four states reported challenges related to the “slow legislative process” (44%), while three (33%) reported “staff turnover” and “resistance to change” as key challenges.

General Observations and Conclusions State collaborative teams that made decisions regarding which aspects of their integrated system to focus on were more successful. This process began with the development of the TAP and subsequent VQ sessions. Contiguous Technical Assistance Plan: Stats that achieved Pending Change or Systems Change were more likely to create what we refer to as contiguous TAPs: Their primary activities were connected in ways that build capacity and lead to systems change from their first milestone to their final milestone. Partitioned Technical Assistance Plan: A partitioned approach includes development of an disconnected activities. States that used a partitioned approach to TAP development were less likely to build toward the goal of systems change.

General Observations and Conclusions Systems change required direct access to executive-level agencies and their leaders. State collaborative teams that had political synergy in terms of their policy objectives with executive-level/agency personnel were more likely to achieve systems change Several state collaborative team leaders described their state’s executive leadership as totally on board with the principles of Employment First, while others reported that their state leaders were not necessarily adverse to expanding access to competitive integrated employment but had not yet made it a priority in their states. Most state leads reported consistent representation in monthly and quarterly meetings. Team members appeared to be engaged, respected, and integrated into the process of building capacity and systems change as well as committed to the milestones included in their TAPs.