1. The webinar, “SAP/ICC Stakeholder Involvement in Evaluating the State Systemic Improvement Plan (SSIP)” will start at 2:30 p.m. While you wait, consider introducing yourself and responding to any of the following questions in the chat space: – What are some of the challenges your State has faced in planning for evaluation? – What are some successes or “aha moments” your State has had when planning for evaluation activities? – What is your understanding of the value that stakeholders bring to the evaluation process?

2. State Advisory Panel & Interagency Coordinating Council SAP/ICC Stakeholder Involvement in Evaluating the State Systemic Improvement Plan (SSIP) Friday, February 19, 2016

3. Agenda Welcome Logistics Conversation with Office of Special Education and Rehabilitative Services – Michael Yudin, Assistant Secretary OSERS – Ruth Ryder, Acting Director, Office of Special Education Programs (OSEP) Presentations – Stakeholder Involvement in Evaluating SSIP – IDEA Partnership – North Carolina – Louisiana Questions (time permitting) Closing Comments and Announcements

4. Your Experiences with Evaluation What are some of the challenges your State has faced in planning for evaluation? What are some successes or “aha moments” your State has had when planning for evaluation activities? What is your understanding of the value that stakeholders bring to the evaluation process?

5. SAP/ICC Stakeholder Involvement in Evaluating the State Systemic Improvement Plan (SSIP) Stakeholder Developed Rubrics: Tools to Assess and Shape Practice – Luz Hernandez, HUNE – John McLaughlin, IDEA Partnership at NASDSE North Carolina – Beverly Roberts, ECAC – Heather Reynolds, NC DPI Louisiana – Brenda Sharp, LA Early Steps – Chris Cedotol, ICC – Nina Seneca, ICC

6. Stakeholder Developed Rubrics: Tools to Assess and Shape Practice

7. Stakeholder Engagement Core Principle o Results Driven Accountability (RDA) o State Systemic Improvement Plan (SSIP) Help describe the ‘principle’ in action from the perspective of the stakeholders

8. Our Theory of Change If... we meaningfully engage stakeholders in the every aspect of the evaluation from selecting the program to be evaluated, evaluation design, implementation, reporting, and use... Then... the evaluation will have a greater probability of producing useful information to confidently communicate success and make necessary improvements where appropriate, leading to impactful and sustainable change.

9. The Purpose of the Evaluation Rubric NOT to teach users how to evaluate their programs ! Rather to identify places at which stakeholders might best be integrated into the process and how to effectively achieve meaningful engagement. Our assumption is that there will be evaluation expertise available to the evaluation team.

10. The Purpose of the Evaluation Rubric Our aim is to help SEA staff and stakeholders become partners in good evaluation. o The Program Evaluation Standards and Guiding Principles provide a framework for creating an effective evaluation and assessing the quality of evaluation planning and implementation. o Focus: program design, program installation/implementation, and program effects. The Rubric presents a set of evidence-based practical tools to enhance stakeholder engagement throughout the evaluation process.

11. Operational Decisions Key actions and behaviors that you will pay attention to Informing Level Sharing/Disseminating- One-way communication Networking Level Exchanging- Two-way communication Collaborating Level Engaging- Working together on the issue over time Transforming Level Committing to approach this and other issues through engagement and consensus building Stakeholder participation as an underlying value in evaluation Convener/state lead agency outlines the evaluation goal and process. They commission an external evaluation and inform the stakeholders that an evaluation is underway. The convener/state lead agency invites a core group to review and give input on the evaluation design, focus and process. They have a dialogue around the evaluation. Convener/state lead agency together with an expanded group of stakeholders build understanding of the goals and use of evaluation. Stakeholders reach agreement on evaluation measures. There is an expectation that evaluation is embedded in the work. Convener/state lead agency ensure that people who are most impacted in the evaluation results are most engaged in the evaluation process. T here is an expectation that stakeholders are partners in evaluation. Evaluation practice addresses knowledge that resides with practitioners and consumers Convener/state lead agency describe the data, grounding assumptions, theory of action and logic model behind the evaluation plan. Convener/state lead agency discusses data, theory of action, logic model, activities and measures designed to inform the evaluation with a core group of stakeholders. Convener/state lead agency and an expanded group of stakeholders examine the data to develop the theory of action, logic model, activities and measures. They meet frequently enough to determine if the theory, logic, activities and measures are working to create practice change. Convener/state lead agency ensures that are stakeholders are always partners in evaluation. Those with the most to lose or gain are involved shaping and acting on evaluation information to improve the system. Problem of Practice: Stakeholder Engagement in Evaluation

16. Why Engage Stakeholders in the Evaluation? We believe that evaluation should be done with people rather than to them. Evaluation emphasizes program implementation and participants' experiences. Stakeholder voices matter for both of these goals. State agency staff are charged with many responsibilities. They need help. Engaging stakeholders is a way to obtain additional resources to support the evaluation project.

17. Why Engage Stakeholders in the Evaluation? Involving stakeholders during evaluation planning and implementation can: o add value by providing perspectives on what will be considered a credible, high quality and useful evaluation o contributing to the program logic and framing of key evaluation questions, o facilitating quality data collection, o help to make sense of the data that has been collected, o increase the utilization of the evaluation’s findings by building knowledge about and support for the evaluation. Stakeholders help the SEA and evaluators understand different perspectives on what will be considered credible evidence of outcomes and impacts.

18. Final Thoughts Evaluation goes beyond performance measurement to get to the why of performance. Different Audiences for Evaluation o Program implementers to inform them of what’s working and not working so they can do something about it. o Funders and policy makers so they can have confidence in the claims that are made about the program o Program participants – it is their right to be exposed to a relevant, high quality program that meets their needs and, in the end, evaluation serves this purpose.

19. NC SSIP & Stakeholder Partnership

20. Contact Information Beverly Roberts, Exceptional Children’s Assistance Center broberts@ecacmail.org Heather Reynolds, NC Department of Public Instruction heather.reynolds@dpi.nc.gov

21. SICC Strategic Plan and State System Improvement Process (SSIP ) State Interagency Coordinating Council

22. Louisiana’s System Design Approach: A Historical Perspective Early Part C (Part H) Days  During the planning years, SICC members established committees to address the design work for the early intervention system  Stakeholder involvement was a key feature of the design  Strategic planning with stakeholders has always been a feature of the system—SICC developed their own plan Current times  In 2008, the Lead Agency and the SICC jointly developed a strategic plan with improvement activities for the lead agency and the ICC  Committees were developed to address the work scope: Professional Development, Program Components, System Resources, and Public Relations  Committees include staff and stakeholders

23. SSIP Planning: SICC Members and Stakeholders gather to discuss Indicator 11 Assemble Review Available Data: APR, Early Childhood Redesign, Strategic Plan and OCDD Transformation Plan Review Data Look for areas where initiatives align/overlap Align State Initiatives What do we need to gather that we don’t have? Look for trends and areas for improvement. Create Workgroups How we organized for the SSIP: Phase I State Leadership Team

24. Four areas were reviewed: The SICC Strategic Plan The Early Care and Education Redesign Framework The Developmental Disability System Transformation Plan The requirements of the SSIP in proposed Indicator 11 Reengage stakeholders in looking for a focus for system improvement

25. Workgroup 1: Aligns with Early Childhood Care and Education System Strategy for Success # 4: Resources and support to improve quality of care and instruction. Group 1: Strategic Direction 2 —Enhancing Services and Accountability Goal 1: Develop a PD System based on evidence-based and promising practices The EarlySteps system will include Family/Professional training that incorporates partnerships that result in quality services with positive outcomes for children and families.  Known as the Professional Development Group

26. Workgroup 2: Aligns with Early Childhood Initiative Strategy: Families have information and access to high quality choices.  Group 2: Strategic Direction 1—Improving the System: Coherent Improvement Strategies:  Objective 5: Through the use of effective team-based decision making, early intervention resources will be more effectively identified and available to meet child and family concerns and priorities without regard to area of residence in the state, resulting in improved child outcomes.  Objective 6: Through the use of improved professional development and alternative service delivery options, children and families will have improved access to needed early intervention supports  Known as the Resource Availability Workgroup

27. Workgroup 3:  Group 3: Known as the Early Childhood Outcomes Workgroup  Goal: Develop implementation Strategies to improve the Early Childhood Outcomes Process to align with the Early Care and Education System. Coherent Improvement Strategies:  Objective 1: Through improved alignment of EarlySteps with the Act 3 Early Care and Education system, increased numbers of infants and toddlers with disabilities will be identified and supported in early care and education settings  Objective 2: Through an improved measurement process, there will be an increase in the number of infants and toddlers exiting the program at or above the level of their typical peers.  Objective 3: Through development and implementation of IFSP outcomes which reflect family concerns, priorities and resources, there will be an increase in the number of infants and toddlers exiting the program at or above the level of their typical peers.

28.  Set up a format using the strategic plan format and added items to meet a logic model format: objectives, activities, measures of success, timelines, inputs/resources available, resources needed  Update/provide detail for improvement  Strategies  Develop the outcome measures How we organized for the SSIP: Phase II Evaluation

29. Questions? Please use the chat area to type in your questions.

30. Thank You! Please evaluate this webinar by going to: https://www.surveymonkey.com/r/SAP21916 If you are the official contact for your State’s ICC or SAP, you can update your contact information by sending an email to info@stateadvisorypanel.org 30