Evaluation Workshop 10/17/2014 Presenters: Jim Whittaker – KCP/Evaluation Dr. Larry M. Gant – SSW Evaluation Christiane Edwards – SSW Evaluation 1

Session Overview  Goal: Continuous Improvement  Objective 1. Identifying Evaluation Strategies to Establish Evidence Based Practices  Objective 2. Moving Activities from Emerging to Best Practice through Evaluation  Objective 3. Communication – Sites, State, and SSW  Wrap-Up  Evaluation Round Table  Walk away with evaluation tools to take your work to the next level. 2

Evaluation – Data Collection – Assessments – Best Practices – Methodology?????????? 3

Evaluation Goals  Short Term Goals. Within the next six months, the Evaluation Team will:  Provide technical assistance to University Program Partners in capturing service data within MI TRACKR.  Provide intensive regional MI TRACKR training sessions on quarterly basis.  Format data and/or data files -create a unified roster to prepare data for analysis, -conduct an initial analysis of program service data from 8 th and 9 th grade cohort. 4

Short-Term Goals continued  Continuously update the MI GEAR UP website with relevant information, in particular, materials to support MI GEAR UP Program Partner evaluation strategies such as toolkits, templates, workshop resources and other relevant materials.  Provide technical assistance regarding: -Identifying Evaluation Strategies to Establish Evidence Based Practices. -Moving Activities from Emerging to Best Practice through Evaluation. -Communication of Best Practices  Assist MI GEAR UP Program Partners in developing evaluation and survey instruments directly aligned to service delivery objectives and goals. 5

Longer Term Evaluation Goals  Develop a detailed Evaluation Plan that directly reflects the state evaluation research questions.  Complete a detailed analysis of MI GEAR UP services in relationship to MI GEAR UP goals, objectives, and outcomes.  Develop a strategic plan for continuous program improvement through evaluation strategies  Publish academic articles to promote MI GEAR UP Best Practices. 6

Objective 1. Identifying Evaluation Strategies to Establish Evidence Based Practices  Review Emerging to Best Practice components.  Identify real life evaluation strategies  Evaluate these strategies and apply them to your MI GEAR UP programs.  Select one of your signature practices as either an Emerging, Promising or Best Practice.  Identify missing components. 7

Objective 1. Identifying Evaluation Strategies continued  Is your practice replicable? -Is there a lesson plan or template?  Is your practice supported by strong qualitative and quantitative data showing positive outcomes? -Is there a survey or other evaluation instrument to provide the data? 8

Emerging  Promising  Best Practices Incorporates qualities of other positive/effective interventions. Is based on patterns that have been proven to lead to effective outcomes. Incorporates a process of continual quality improvement Has evaluation plan in place Emerging Practice In addition to the qualities of an emerging practice, has strong qualitative and quantitative data showing positive outcomes Promising Practice Has undergone a rigorous process of peer review and evaluation that indicates effectiveness Best Practice Source: “Emerging, Promising, and Best Practices Definitions” from the Kentucky Cabinet for Health and Family Services 9

Identify Real Life Evaluation Strategies  Learn to bake a delicious Whipping Cream Pound Cake  Can you replicate this pound cake with the information given in the clip? If not, what is missing? 10

Identify Real Life Evaluation Strategies  Have you ever booked a hotel room, bought a house or a car?  Identify evaluation strategies you would use to determine which hotel room, house or car to select? 11

Identify Real Life Evaluation Strategies (continued)  Are you using a Cost Benefit Analysis or Criterion Based Matrix to assess whether a purchase is worth your money? a) Cost vs. Location b) Cost vs. Size c) Cost vs. Benefits d) Cost vs. Amenities e) Cost vs. Appearance By using these strategies, you are actually conducting a small scale evaluation! NOW, LET’S TAKE THIS EXAMPLE AND APPLY IT TO MI GEAR UP. 12

Objective 1. Identify Evaluation Strategies to Establish Evidence-Based Practices.  Like a recipe, your lesson plan serves as the tool or blue print for replication.  A lesson plan is one of the components for establishing a Best Practice.  If your practice is replicable, it can reliably produce strong qualitative and quantitative data supporting positive outcomes. 13

Objective 1. Identify Evaluation Strategies to Establish Evidence-Based Practices. Using a Criterion Based Matrix or Cost Benefit Analysis for your practice can demonstrate significant positive outcomes. Program Services Delivery Time & Money (PSDT & M) vs. the following program goals:  Increased understanding of objectives and goals  Increased GPA  Increased standardized test scores  Increased understanding of college affordability  Increased FAFSA completion rates  Promotion to the next grade level  Increased graduation rates Information from the analysis will allow you to continuously improve the variables to achieve better outcomes. 14

Evaluation Program Workshop: Emerging/Promising/Best Practice  Please place your practice on the Emerging/Promising/ Best spectrum. 1.Identify which the practice components you currently have in place. 2.Identify which the practice components you would like to establish. 3.During the roundtable discussion, you may share which evaluation strategies you are currently using for continuous program improvement. 15

Objective 1 Worksheet. Identifying Evaluation Strategies to Establish Evidence-Based Practices Program Partner:__________________________________________ Signature Practice: ________________________________________ Directions: Place Your Practice on the Emerging/Promising/Best Practice spectrum. Use information from your previously completed Characteristics of Promising/Best Practices document to complete the following: 1.Identify which of the seven Promising/Best Practice components you currently have in place and describe how those components apply to your practice. 2.Identify which of the seven Promising/Best Practice components you would like to establish and describe the necessary actions steps to implement them. 16

Take Your Selected Practice and Use It for Continuous Program Improvement. 1.A Promising Practice requires strong qualitative and quantitative data showing positive outcomes. Acquiring this data involves both a lesson plan for replicability and an evaluation tool to demonstrate positive outcomes. 2.Identify the components of a lesson plan to support the replicability of your practice. 3.Identify the components of an evaluation instrument to support qualitative and quantitative data supporting the activity as a promising practice. 17

Objective 2. Moving Activities from Emerging to Best Practice through Evaluation  Moving from Emerging to Best Practices is an ongoing activity. The Evaluation Team is available to provide the following assistance:  Development of lesson plan templates  Draft survey instruments and other evaluation tools 18

Objective 2 Worksheet. Moving Activities from Emerging to Best Practice through Evaluation. Program Partner: ________________________________ Signature Practice: _______________________________ Directions: A lesson plan is one of the components for establishing a Best Practice. If your practice is replicable, it can reliably produce strong qualitative and quantitative data supporting positive outcomes. To effectively replicate a practice, a lesson plan should include the following components:  Goal: What the student will have learned by end of the activity.  Objective: Measures for whether the goal has been achieved  Activity: Step-by-step description of the practice  Assessment: Measures whether goal has been achieved 1.Identify the lesson components for your signature practice: 19

Objective 2 Worksheet. Moving Activities from Emerging to Best Practice through Evaluation. Program Partner: ________________________________ Signature Practice: _______________________________ Directions: Once you have a replicable practice, you will want to develop and implement an evaluation tool to capture qualitative and quantitative data supporting positive program outcomes. 2.Identify an evaluation tool to capture data from your practice and describe how the evaluation would be implemented: 20

Objective 3. Communication of Evaluation Results  Sites  Ongoing program improvement  Share program success with local stakeholders  State  Share program success with national and state stakeholders  Inform programmatic decisions and resource allocation  SSW  Publish program outcomes. 21

Action Steps 1.Identify one to two signature program services that will be transformed to a Best Practice. 2.Continuously improve those services through evaluation strategies. 3.Develop and implement the lesson plan for replicability. 22

Action Steps 4.Develop evaluation tools, in cooperation with the Evaluation Team, to support your practice with quantitative and qualitative data. 5.Communicate your improvements and signature work with the Evaluation Team, so that the work may be published. 23

Wrap Up 24

Evaluation Roundtable Discussion  Handout for Discussion Questions is provided. 25