1. Leveraging Evaluative Metrics And Developing Tactical Interventions that Maximize Program Impact: A Real World Example Presented October 27, 2012 26th Annual Conference of the American Evaluation Association in Visualizing Collaboration and Impact through Social Network Analysis and Perceptual Maps Jack Fentress, M.S., M.B.A. Herbert M. Baum, Ph.D.

2. Background  History—assessments to quantify program impact  Methodology—pre/post assessment, 20 items, five-point Likert scale  Objective 2 Evaluative (Measure Impact) Tactical (Improve)

3. Statement of the Problem  Muted program impact scores  Kids who have heard some of this before  Programs in need of clear tactical feedback Typical solutions for a standard Pre/Post evaluation design mean score, top 2 box score 3 Standard Tools

4. Ceiling Effect 4

5. ISSUE #1: Muted Impact Scores 5

6. Receptive - I am willing to listen to other people's ideas even if they are different from my own Clarify - I ask questions when people say something I do not understand Express - I am good at getting my point across so others understand what I mean Adaptable - When things change I can adjust my plans Plans - You need to have a plan to get to your goal Goals - It is important to set goals for yourself Contribute - I can help a team be successful Respect - Teams work better when people treat each other with respect Resolution - I know some positive ways to settle disagreements when working in a team Evaluate - I can usually look at a situation in a number of ways Realistic - It is important to think about possible good and bad results when making a choice Responsible - I am responsible for the decisions I make Persistent - I can solve difficult problems if I continue to try Reflection - You can learn from mistakes to make better choices the next time Leader - If people were choosing someone to lead an activity, they might choose me Effective - I can do a lot when I set my mind to it Proud - Considering everything, I have a lot to be proud of Impactful - My own hard work can change things for the better Resourceful - I can handle things even when something unexpected happens Independent - I can stand up for what I believe is right even when the group does something different Communication Planning Avg. Impact for Item = 7.0% Range 0% – 14% 6

7. Receptive - I am willing to listen to other people's ideas even if they are different from my own Clarify - I ask questions when people say something I do not understand Express - I am good at getting my point across so others understand what I mean Adaptable - When things change I can adjust my plans Plans - You need to have a plan to get to your goal Goals - It is important to set goals for yourself Contribute - I can help a team be successful Respect - Teams work better when people treat each other with respect Resolution - I know some positive ways to settle disagreements when working in a team Evaluate - I can usually look at a situation in a number of ways Realistic - It is important to think about possible good and bad results when making a choice Responsible - I am responsible for the decisions I make Persistent - I can solve difficult problems if I continue to try Reflection - You can learn from mistakes to make better choices the next time Leader - If people were choosing someone to lead an activity, they might choose me Effective - I can do a lot when I set my mind to it Proud - Considering everything, I have a lot to be proud of Impactful - My own hard work can change things for the better Resourceful - I can handle things even when something unexpected happens Independent - I can stand up for what I believe is right even when the group does something different Communication Planning Avg. Impact for Item = 9.7% Range 2% – 20% 7

8.  Change calculated at subject level  Uses all response positions on the scale An Alternative Approach Subject-Level Calculations 8  Redefines “Impact”—Accounts for movement in positive direction (e.g., Strongly Disagree to Disagree, Agree to Strongly Agree) and subtracts out movement in negative direction (e.g., Agree to Neutral) Net Gain % = Better rating % - Worse rating %  Shows this is more sensitive to change than either the change in mean score or the change in Top 2 Box score Strongly disagree Disagree Neutral Agree Strongly agree How much do you agree with the following statements? Its important to set goals for yourself.

9. Calculation Example: 9 SubjectPre-TestPost-TestChange 134+ 245+ 3440 424+ 554- Net Gain Mean3.64.2+17% T2B35+40% Subject Level+40%

10. Receptive - I am willing to listen to other people's ideas even if they are different from my own Clarify - I ask questions when people say something I do not understand Express - I am good at getting my point across so others understand what I mean Adaptable - When things change I can adjust my plans Plans - You need to have a plan to get to your goal Goals - It is important to set goals for yourself Contribute - I can help a team be successful Respect - Teams work better when people treat each other with respect Resolution - I know some positive ways to settle disagreements when working in a team Evaluate - I can usually look at a situation in a number of ways Realistic - It is important to think about possible good and bad results when making a choice Responsible - I am responsible for the decisions I make Persistent - I can solve difficult problems if I continue to try Reflection - You can learn from mistakes to make better choices the next time Leader - If people were choosing someone to lead an activity, they might choose me Effective - I can do a lot when I set my mind to it Proud - Considering everything, I have a lot to be proud of Impactful - My own hard work can change things for the better Resourceful - I can handle things even when something unexpected happens Independent - I can stand up for what I believe is right even when the group does something different Communication Planning Avg. Impact for Item = 22.8% Range 4% – 33% 10

11. Receptive - I am willing to listen to other people's ideas even if they are different from my own Clarify - I ask questions when people say something I do not understand Express - I am good at getting my point across so others understand what I mean Adaptable - When things change I can adjust my plans Plans - You need to have a plan to get to your goal Goals - It is important to set goals for yourself Contribute - I can help a team be successful Respect - Teams work better when people treat each other with respect Resolution - I know some positive ways to settle disagreements when working in a team Evaluate - I can usually look at a situation in a number of ways Realistic - It is important to think about possible good and bad results when making a choice Responsible - I am responsible for the decisions I make Persistent - I can solve difficult problems if I continue to try Reflection - You can learn from mistakes to make better choices the next time Leader - If people were choosing someone to lead an activity, they might choose me Effective - I can do a lot when I set my mind to it Proud - Considering everything, I have a lot to be proud of Impactful - My own hard work can change things for the better Resourceful - I can handle things even when something unexpected happens Independent - I can stand up for what I believe is right even when the group does something different Communication Planning Avg. Item Change Mean Score 7% Top 2 Box Score 10% Subject Level 23% Avg. Item Change Mean Score 7% Top 2 Box Score 10% Subject Level 23% 11

12. ISSUE #2: Programs needing clear tactical guidance. 12

13. Communicating Tactics–Perceptual Maps  Juxtaposes Status and Impact  Status represented by Pre-Assessment Mean Score  Growth represented by Net Gain % Score  Use to identify tactical opportunities 13

14. StrengthsOptimized Weaknesses Inconclusive Perceptual Maps 14

15. StrengthsOptimized WeaknessesInconclusive 15 StrengthsOptimized Weaknesses Inconclusive

16. 16 StrengthsOptimized Weaknesses Inconclusive

17. Key Take-Aways  Impact can be amplified and better quantified with subject-level analyses.  Matching scores will require some operational planning.  Subject-level results afford new possibilities— segmentation and simulations.  Perceptual maps can effectively communicate tactical guidance. 17

18. Thank You! For Additional Information Jack Fentress JFentress@DataRecognitionCorp.com or Herb Baum HBaum@DataRecognitionCorp.com 18