Three Levels of Project Evaluation SPDG Evaluators PLC Anna Harms December 11, 2012

2 Systems and Structures for Supporting Evaluation Three Levels of Evaluation Agenda

Feedback Loop to Determine Worth Management/CoordinationManagement/Coordination Information Resources PeopleMaterialsCapitol Work Systems: Providing the information and materials to implement MTSS practices and the supports for the these practices to occur successfully within schools and districts Investments: Funding Visibility Political support Investments: Funding Visibility Political support Returns: Addressing critical issues Addressing program directives Returns: Addressing critical issues Addressing program directives Consumers students families schools, districts, ISDs Consumers students families schools, districts, ISDs Outputs: MTSS Products PD & TA services Outputs: MTSS Products PD & TA services FinancialEvaluationProfessional Learning Technical Assistance MiBLSi Value Added Logic Model Provides guidance, problem solving, and feedback on organizational infrastructure/ca pacity to support implementation of MTSS Provides skill/knowledge develop with information and materials to support individual competencies in the implementation of MTSS Examines project activities/output s for improving efficiencies and project outcomes for efficacy of implementation of MTSS Allocation of funds and dissemination of funding opportunities for implementation of MTSS Impact/Outcomes Short Term: Change Learning Intermediate Term: Change Behavior Long Term: Change Conditions Stakeholders & Funders Political Regulatory/Legal Economical Geographical Societal/Culture Political Regulatory/Legal Economical Geographical Societal/Culture Environmental Influences

4 MiBLSi Organizational Structure

5 A combination of internal and external evaluation Communities of support A data-oriented organizational culture What helps us accomplish the evaluation work?

6 A Disclaimer:

7 Evaluation Systems LevelDescriptionTools OrganizationalEvaluating impact of the MTSS project Stakeholders Addressing critical issues (e.g., Discipline/Ethnicity) Addressing program directives (e.g., State Performance Plan) Consumers Valued results (e.g., Improved quality program, fidelity of implementation, successful student outcomes) Annual reports Measures of program quality and implementation fidelity Measures of student outcomes Surveys of consumer satisfaction ProcessEvaluating operational productivity and efficiency of each unit within organization, as well as inter-unit collaboration and communication Project management tools Smartsheet Dashboards Annual Interviews Individual Worker Evaluating worker productivity and efficiency Job models and staff evaluation procedures

Develop Organizational Structures Develop Staff Competencies Schools effectively implement MTSS Practices Successful Student Outcomes Outcomes Inputs Feedback Loops

9 Include: Training Technical Assistance Are measured through: End of Training Evaluation Data Inputs

10 End of Day Evaluations

11 Retrospective Self Assessment

Develop Organizational Structures Develop Staff Competencies Schools effectively implement MTSS Practices Successful Student Outcomes Outcomes Inputs Feedback Loops

13 Intermediate School District Implementation Fidelity

Met SAS Schoolwide Criteria-70% (36 schools) Did not meet SAS Schoolwide Criteria- 70% (30 schools) National Median

Develop Organizational Structures Develop Staff Competencies Schools effectively implement MTSS Practices Successful Student Outcomes Outcomes Inputs Feedback Loops

16 Consumer Feedback Survey

17 Do you support the organizational level of evaluation for your state’s SPDG? YesYes NoNo What are some notable similarities and differences when comparing your organizational level evaluation with what has been presented? Feedback

18 Evaluation Systems LevelDescriptionTools OrganizationalEvaluating impact of the MTSS project Stakeholders Addressing critical issues (e.g., Discipline/Ethnicity) Addressing program directives (e.g., State Performance Plan) Consumers Valued results (e.g., Improved quality program, fidelity of implementation, successful student outcomes) Annual reports Measures of program quality and implementation fidelity Measures of student outcomes Surveys of consumer satisfaction ProcessEvaluating operational productivity and efficiency of each unit within organization, as well as inter-unit collaboration and communication Project management tools Smartsheet Dashboards Annual Interviews Individual Worker Evaluating worker productivity and efficiency Job models and staff evaluation procedures

19 Task Completion: Professional Learning Unit

20 Dashboard: Evaluation and Research Unit

21 Feedback on: Effectiveness of the organizational structure Annually in the Spring Individual Interviews / Surveys Unit / Team Interviews Summary Report and Debrief Interviews with External Evaluator (Dr. Julie Morrison)

22 Do you support the process level of evaluation for your state’s SPDG? YesYes NoNo What are some notable similarities and differences when comparing your process level evaluation with what has been presented? Feedback

23 Evaluation Systems LevelDescriptionTools OrganizationalEvaluating impact of the MTSS project Stakeholders Addressing critical issues (e.g., Discipline/Ethnicity) Addressing program directives (e.g., State Performance Plan) Consumers Valued results (e.g., Improved quality program, fidelity of implementation, successful student outcomes) Annual reports Measures of program quality and implementation fidelity Measures of student outcomes Surveys of consumer satisfaction ProcessEvaluating operational productivity and efficiency of each unit within organization, as well as inter-unit collaboration and communication Project management tools Smartsheet Dashboards Annual Interviews Individual Worker Evaluating worker productivity and efficiency Job models and staff evaluation procedures

24 Practice Profiles for Each Position

25 Self Assessment Consumer Feedback Observation Supervisor Feedback Used through a cyclical process of goal setting, data analysis, celebrations and problem solving. Data Sources

26 Do you support the individual worker level of evaluation for your state’s SPDG? YesYes NoNo What are some notable similarities and differences when comparing your individual worker level evaluation with what has been presented? Feedback

27 Thank you! What feedback, questions, and ideas do you have? Feedback

Contact Information Anna Harms Evaluation and Research Coordinator Michigan’s Integrated Behavior and Learning Support Initiative (616)