1. EFLSMP Summative Evaluation
Social Dynamics, LLC
481 North Frederick Ave., Suite 410
Gaithersburg, MD 20877
Phone: (301)
EFLSMP Summative Evaluation
Douglas Klayman, Ph.D.
September 2016

2. Overview Program Characteristics State Topics Program Components
Summative Evaluation Methodology
Data Sources
Measuring Systems Change and Increases in Capacity
General Observations and Conclusions
Employment First is really a framework for systems change that is centered on the premise that all citizens, including individuals with significant disabilities, are capable of full participation in integrated employment and community life
Integrated, Competitive Employment (ICE): Jobs that people want; are located in competitive, integrated settings; and that pay at least the federal minimum wage
EFSLMP assists states in may changes in their systems and capacity to provide competitive integrated employment. One of the key approaches is rate restructuring along with provider transformation pilot projects.

3. Program Characteristics
Concepts
Federal Initiative
Employment First: A framework for systems change that is centered on the premise that all citizens, including individuals with significant disabilities, are capable of full participation in integrated employment and community life
Integrated, Competitive Employment (ICE): Jobs that people want; are located in competitive, integrated settings; and that pay at least the federal minimum wage
Employment First State Leadership Mentoring Program (EFSLMP): The vehicle for implementing
Systems Change, Capacity Building and states’ ability to Transform employment services from sheltered/day programs to ICE
The various tiers of technical resources offered under EFSLMP have expanded over the years.
Vision Question
Three-tiered program:
Tier 1 (SME-facilitated T/TA services) and Tier 2 (VQ strategic and organizational planning activities) are offered to all core states while Tier 3 (Community of Practice) is available to any interested stakeholder (advocate, direct support professional, provider, government official, and others) in all 50 states and the District of Columbia. The sample of nine core states that are included in the 2015 summative evaluation had direct access to all three EFSLMP resource tiers.
In 2015, the CoP provided webinars on a range of policy and practice innovations, such as benchmarking, development and maintenance of governing boards, provider transformation and its impact on staff, customized employment from the employer’s perspective, engagement of employers in the recruitment and employment of individuals with significant disabilities, competitive integrated employment, IPS, staff competencies for customized employment and self-employment, decentralization of service delivery systems, pre-vocational services and WIOA provisions.

4. Program Characteristics cont’d
Federal Employment Systems Change and Capacity Building Initiative
Employment First State Leadership Mentoring Program (EFSLMP): The vehicle for implementing Systems Change, Capacity Building and states’ ability to Transform employment services from sheltered/day
Systems change related to the policy landscape in support of competitive integrated employment
EFSLMP assists states in identifying and addressing changes in state policies, funding models, and practices to increase access to and outcomes associated with competitive integrated employment.
Employment First is really a framework for systems change that is centered on the premise that all citizens, including individuals with significant disabilities, are capable of full participation in integrated employment and community life
Integrated, Competitive Employment (ICE): Jobs that people want; are located in competitive, integrated settings; and that pay at least the federal minimum wage
EFSLMP assists states in may changes in their systems and capacity to provide competitive integrated employment. One of the key approaches is rate restructuring along with provider transformation pilot projects.

5. State Topics
Provider Transformation
Director Support Prof Capacity Building in Effective Practices
School-to-Work Transition
Employer Engagement Strategies
Innovative Payment Schemes & Rate Restructuring
Leveraging & Coordination of Policy and Resources across Systems
Rate restructuring continues to be a very important issue for many states. Combined with provider transformation are the two topics that were consistently part of the program.
School to work transition, Direct Support effective practices, employer engagement and leveraging policy and resources.

6. Other Topics Additional topics covered: Cross-systems change
Implementation of promising practices
Individual Placement and Support
Performance measurement
Policy development and guidance
Effective hiring practices for employers
Employer engagement in the recruitment and employment of individuals with disabilities
Development of executive teams for cross-systems collaboration
WIOA policy implementation and support and development of HCBS waivers and modifications

7. Program Components Tier 1: Intensive Training and Technical Assistance
Purpose:
To provide states with hands-on T/TA that supports the implementation of EF principles
Availability:
Core States
Scope:
Narrow (topics selected by state team leaders)
Frequency:
Varies (2–4 time per year)
Facilitator:
SME
Result:
SME works with selected members of the state team, state, and/or provider staff to provide information (e.g., provider transformation, policy analysis, advocacy, skill building)
Tier 2: Vision Quest
To engage states in developing strategies for resolving challenges related to EF implementation
Moderate (topics selected by state team)
Quarterly
State team works with SME to resolve challenges
Tier 3: Community of Practice
To learn about promising practices and strategies for EF implementation
All states
Broad (topics selected by ODEP and SMEs)
High (monthly)
SME or guest presenter
Promising practices can be adopted by EF states for implementation
There are three level of T/TA.

8. Summative Evaluation Methodology
Objective:
Identify systems change and increases in capacity to provide competitive integrated employment opportunities for individuals with significant disabilities
Qualitative Data Collection: Collection and coding of systems change and capacity building products.
Sample:
Purposive sampling methodology for heterogeneity used to select nine core states out of the universe of 15.
Geographically representative of the universe of the EFSLMP core states and chosen.

9. Data Sources
Baseline and follow-up interviews with state collaborative team leaders
60-90 minute stakeholder interviews
Detailed reviews of core state Technical Assistance Plans (TAP)
Assessments of systems change and capacity-building products
Reports and recordings s from Vision Quest (VQ) sessions
T/TA webinars
Professional development trainings provided by subject matter experts
Monthly core state reports,
We used baseline and two follow-up interviews to collect detailed information. This list is really in order of the importance of the information we collected. The protocol was aligned with the technical assistance plan (TAP). Let me show you an example of the TAP. It was really the key document that facilitated our data collection approach.

10. Measuring Systems Change and Increases in Capacity cont’d
ODEP’s Criteria for Performance Excellence in Employment First State Systems Change and Provider Transformation
Three systems change and two capacity building factors:
1. Monitoring and reacting to changes in demand among individuals with significant disabilities for competitive, integrated employment services (systems change factor);
2. Use of evidence-based practices (capacity building factor);
3. Changes in legislation/policy that positively affects individuals with disabilities (systems change factor);
4. Improvements in cost-effectiveness of services (systems change factor);
5. Use of rigorous performance measurement procedures that include key metrics such as employment placement and retention outcomes and receipt of prevailing wages (capacity building factor).
This slide shows the systems change framework. It was developed by ODEP and adopted by us as a way to determine if the product (or document) focused on systems change and/or capacity building. For example, a product that documents changes in demand among individuals with disabilities for competitive, integrated employment, we categorized as a systems change product. One that focuses on evidence based practices and the use of new practices, we categorized as a capacity building factor. Performance measurement procedures was also a capacity building factors. So, for example, if we are concerned with the first factor, which is a systems change factor, we directed our research on each product, including interviews with stakeholders, to give us information on how a given product, increased demand for integrated employment. If it was a capacity building product, we would ask questions of respondents related to its ability to increase capacity for competitive integrated employment. For every product we had, and there were many, they were categorized based on one or more of these factors.

11. Measuring Systems Change and Increases in Capacity cont’d
Three systems change and two capacity building factors:
1. Systems Change Factor 1: Monitoring/Reacting to Changes in Demand
2. Capacity Building Factor 1: Evidence-Based Practices
3. Systems Change Factor 2: Changes in Legislation/Policy
4. Systems Change Factor 3: Improvements in Cost-Effectiveness of Services
*5. Use of rigorous performance measurement procedures that include key metrics such as employment placement and retention outcomes and receipt of prevailing wages (capacity building factor).
This slide shows the systems change framework. It was developed by ODEP and adopted by us as a way to determine if the product (or document) focused on systems change and/or capacity building. For example, a product that documents changes in demand among individuals with disabilities for competitive, integrated employment, we categorized as a systems change product. One that focuses on evidence based practices and the use of new practices, we categorized as a capacity building factor. Performance measurement procedures was also a capacity building factors. So, for example, if we are concerned with the first factor, which is a systems change factor, we directed our research on each product, including interviews with stakeholders, to give us information on how a given product, increased demand for integrated employment. If it was a capacity building product, we would ask questions of respondents related to its ability to increase capacity for competitive integrated employment. For every product we had, and there were many, they were categorized based on one or more of these factors.
*Not used by any state in evaluation.

12. Measuring Systems Change and Increases in Capacity cont’d
Examples
Systems Change Factor 1: Monitoring/Reacting to Changes in Demand
Systems Change Factor 2: Changes in Legislation/Policy
Systems Change Factor 3: Improvements in Cost-Effectiveness of Services
Capacity Building Factor 1: Evidence-Based Practices
New Partnership Linking VR
with Workforce
New HCBS Waiver
Rate Reimbursement/
Restructuring
This slide shows the systems change framework. It was developed by ODEP and adopted by us as a way to determine if the product (or document) focused on systems change and/or capacity building. For example, a product that documents changes in demand among individuals with disabilities for competitive, integrated employment, we categorized as a systems change product. One that focuses on evidence based practices and the use of new practices, we categorized as a capacity building factor. Performance measurement procedures was also a capacity building factors. So, for example, if we are concerned with the first factor, which is a systems change factor, we directed our research on each product, including interviews with stakeholders, to give us information on how a given product, increased demand for integrated employment. If it was a capacity building product, we would ask questions of respondents related to its ability to increase capacity for competitive integrated employment. For every product we had, and there were many, they were categorized based on one or more of these factors.
Provider Transformation

13. Measuring Systems Change and Increases in Capacity cont’d
STEP 1: Connect “on the ground” systems change and capacity building to the broad areas defined by ODEP
For example, using the following T/TA topics:
“policy development and alignment” is associated with factors 1 & 3.
“provider transformation designed to upgrade the skills and knowledge of provider staff/provider transformation” is associated with factor 2.
“rate reimbursement & restructuring” is associated with factor 4
*Factor 5, use of rigorous performance measurement procedures, was included only in Delaware’s TAP. Delaware was not selected for the evaluation and therefore, we excluded factor 5 from the 2015 Summative Evaluation.

14. Measuring Systems Change and Increases in Capacity
Factor 1
Factor 2
Factor 3
Factor 4
MATRIX
Selected EFSLMP Product Topics
“Policy development and alignment”
“provider transformation designed to
upgrade the skills and knowledge of
provider staff/provider transformation”
“rate reimbursement & restructuring” √ √ √ √

15. Measuring Systems Change and Increases in Capacity cont’d
STEP 2: Create systems change, capacity building and TAP milestone rubrics (See Separate PDFs Handouts)
Systems change and capacity building rubric has three positions:
No change in capacity or systems
Pending: change that may happen either subsequent to authorization by an executive member of a state agency or the successful completion of a capacity building project or event.
Completed : Products that have been implemented and as a result, transformed policy and practice during the current fiscal year.

16. Measuring Systems Change and Increases in Capacity cont’d
STEP 3: Increases in Capacity: Measured dichotomously due to the limitations of the data and its lack of clearly defined gradations in the capacity building variable.
Capacity defined: A perceptible increase in the capacity of providers within a state to provide services to individuals with disabilities that lead to competitive integrated employment opportunities.
There are no graduations in the way we defined increased capacity.
Products created by each state were rated separately and could be categorized as both systems change and capacity building.

17. Measuring Systems Change and Increases in Capacity
10 Systems Change Products
Systems change products included documents that created state agency collaborations and led to formal partnerships
Memoranda of Understanding
Interagency cooperative agreements
Modifications to existing policies and practices
Agency directives
Sub-regulatory policy guidance
State legislation
Changes to service definitions and rate structures in VR
HCBS waivers and modifications

18. Measuring Systems Change and Increases in Capacity
23 Capacity Building Products
Capacity-building products included resources that provide technical information on EF policies and practices:
Resource guides
Tool kits
Data collection instruments
Training curricula
Fact sheets
State and local conferences
T/TA and provider trainings

19. Measuring Systems Change and Increases in Capacity
10 Systems Change Products and 23 Capacity Building Products

20. General Observations and Conclusions
Many states developed cross-system partnerships and collaborations connecting Medicaid, Rehabilitation, Intellectual and Developmental Disabilities, Workforce Development, Workforce, Mental Health, Education, and VR agencies with each other through their EFLSMP state collaborative teams.
Organizational inertia caused by limitations in fiscal and/or human resources, slowed process of policy acceptance and implementation
Changing the “status quo” and transforming state systems depended on how collaborative state teams used their capacity-building and systems change products to communicate with state leaders, service providers, and other stakeholder groups.

21. General Observations and Conclusions
Most state collaborative teams expedited the policy process. Some were challenged by a an extended period of time waiting for final clearance or authorization from executive leadership.
Four states reported challenges related to the “slow legislative process” (44%), while three (33%) reported “staff turnover” and “resistance to change” as key challenges.

22. General Observations and Conclusions
State collaborative teams that made decisions regarding which aspects of their integrated system to focus on were more successful. This process began with the development of the TAP and subsequent VQ sessions.
Contiguous Technical Assistance Plan:
Stats that achieved Pending Change or Systems Change were more likely to create what we refer to as contiguous TAPs: Their primary activities were connected in ways that build capacity and lead to systems change from their first milestone to their final milestone.
Partitioned Technical Assistance Plan:
A partitioned approach includes development of an disconnected activities.
States that used a partitioned approach to TAP development were less likely to build toward the goal of systems change.

23. General Observations and Conclusions
Systems change required direct access to executive-level agencies and their leaders.
State collaborative teams that had political synergy in terms of their policy objectives with executive-level/agency personnel were more likely to achieve systems change
Several state collaborative team leaders described their state’s executive leadership as totally on board with the principles of Employment First, while others reported that their state leaders were not necessarily adverse to expanding access to competitive integrated employment but had not yet made it a priority in their states.
Most state leads reported consistent representation in monthly and quarterly meetings. Team members appeared to be engaged, respected, and integrated into the process of building capacity and systems change as well as committed to the milestones included in their TAPs.