1. Anita M. Baker Evaluation Services Building Evaluation Capacity Session (3) 8 Data Visualization, Math for Evaluators Bruner Foundation Rochester, New York

2. General Characteristics of Effective Tables and Graphs The table or graph should present meaningful data. The data should be unambiguous. The table or graph should convey ideas about data efficiently. Materials adapted from Gary Klass (Illinois State University), 2002 Oehlert and Weisberg, 2008 (University of Minnesota) Tufte, Edward, Presenting Data and Information, 2010 1 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

3. Be Kind to your Readers  Label everything!  Use the kind of display best suited to your data.  Identify units: numbers, percentages, places, types of people.  Minimize the ratio of ink-to-data (Tufte): de-emphasize the chart, bring out the data.  Use color to your advantage, but make sure your tables and graphs will print in black and white. 2 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

4. Standards of “Graphical Excellence” Edward Tufte  Well-designed presentation of data of substance  Complex ideas communicated with clarity, precision and efficiency  The greatest number of ideas, in the shortest time, in the smallest space, with the least ink 3 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

5. Children Under 18 Living in Poverty, 2008 CategoryNumber*Percent All children under 1815,45120.7 White, non-Hispanic4,85011.9 Black4,48035.4 Hispanic5,61033.1 Asian 53113.3 * In thousands SOURCE: U.S. Bureau of the Census, Income, Poverty, and Health Insurance Coverage in the United States: 2009, Report P60, n. 238, Table B-2, pp. 62-7. Tables, Tables, Tables, Tables, Tables, Tables, Tables, 4 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

6. Thinking About Tables and Figures Tables are organized as a series of rows  and columns . The first step to constructing a table is to determine how many rows and columns you need. The individual boxes or “cells” of the table contain the information you wish to display. 5 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

7. Thinking About Tables and Figures Tables must have a table number and title (be consistent). Where possible, use the title to describe what is really in the table. Table 1: Percent of Respondents Agreeing with Each Item in the Customer Satisfaction Scale. All rows and columns must have headings. It should be clear what data are displayed (n’s, %s) You don’t have to show everything, but a reader should be able to independently calculate what you are displaying. Clarify with footnotes if needed. Use lines and shading to further emphasize data. 6 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

8. Table Basics  Design for purpose and audience  Round!  Organize  Simplify  Add summaries  Good title/labels  Clean layout/proper spacing 7 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

9. General Characteristics of Effective Tables and Graphs The table or graph should present meaningful data. The data should be unambiguous. The table or graph should convey ideas about data efficiently. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 8

10. Meaningful Data in Tables  Use rates, ratios and per capita measures rather than aggregate totals.  Two time points are better than one.  Show change over a meaningful time period (e.g., 5-year change rather than annual change).  Multi-year trends are best presented in time series charts rather than tables.  Show the source of the data. 9 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

11. Unambiguous Data in Tables  Each number in a table should have a precise meaning.  Use titles, headings, and notes to specify the contents of the table cells, rows and columns.  Carefully select measures: e.g., frequencies (counts) vs. percentages, vs. rates (e.g., number per 100,000).  Clearly define numerators and denominators, and distribution decisions. e.g., % of poor who are children (35%) vs. % of children who are poor (21%) US Census Bureau statistics, 2008  Be especially clear when defining change.  Percentage change vs. percentage point change 10 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

12. Totals vs. Rates Murders* in Ten Largest US Cities, 1998 Chicago703 New York633 Detroit430 Los Angeles426 Philadelphia338 Houston254 Dallas252 Phoenix185 San Antonio89 San Diego42 *Number of cases of murders and non-negligent manslaughter Detroit43.0 Chicago25.6 Philadelphia23.3 Dallas23.1 Phoenix15.1 Houston14.1 Los Angeles11.8 New York8.6 San Antonio8.1 San Diego3.5 * Murder and non-negligent manslaughter per 100,000 population Murder Rates* in Ten Largest US Cities, 1998 Source: Statistical Abstract 2000 CD-ROM tables 332; and Bureau of Justice Statistics: http://www.ojp.usdoj.gov/bjs/data/cities92.wk1 10b

13. Efficient Use of Data in Tables Sort data on the most meaningful variable Time always left to right Similar data goes down the columns Highlight important comparisons Don’t force comparisons between two different tables Use consistent formatting across tables Data can be quickly interpreted by the reader. 11 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

14. Rounding!  Use two significant figures where ever possible. City Budget, 2010 = $27,329,681 City Budget, 2010 = 27 million dollars.  Never forget meaningfulness. Life expectancy = 67.14 years.01 year is about 4 days  The one exception is for archival tables. 12 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

15. Uses for Tables  Exploration/Organization  Storage  Communication Adapted from G. Oehlert, rev. by S. Weisberg, School of Statistics, University of Minnesota, 2/2008 13 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

16. When Using Tables to Communicate  Target an audience  Have a goal (tell a story)  Make the story obvious  Be uncluttered  Cause no pain Big picture Trends Comparisons Typical Values Atypical Values 14 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

17. Two Time Points are better Murder Rates in Ten Largest US Cities, 1995-1998 19951998Net Change Detroit47.643.0- 4.6 Chicago30.025.6- 4.4 Philadelphia28.223.3- 4.9 Dallas26.523.1- 3.3 Los Angeles24.511.8- 12.7 Phoenix19.715.1- 4.6 Houston18.214.1- 4.1 New York16.1 8.6- 7.5 San Antonio14.2 8.1- 6.1 San Diego 7.9 3.5- 4.4 * Murder and non-negligent manslaughter per 100,000 population Source: Statistical Abstract 2000 CD-ROM tables 332; and Bureau of Justice Statistics: http://www.ojp.usdoj.gov/bjs/data/cities92.wk1 15

18. Sort on Meaningful Variables Percent of 9-year-olds who watch more than 5 hours of TV per weekday Country% Canada14.9 Denmark 6.0 Finland 6.1 France 5.5 Germany 4.4 Ireland11.8 Italy 9.2 Netherlands12.6 Spain17.5 Sweden 4.7 United States21.5 Country% United States21.5 Spain17.5 Canada14.9 Netherlands12.6 Ireland11.8 Italy 9.2 Finland 6.1 Denmark 6.0 France 5.5 Sweden 4.7 Germany 4.4 Percent of 9-year-olds who watch more than 5 hours of TV per weekday Source: Uri Bronfenburger, et. al. The State of Americans (New York: The Free Press, 1996) from: William Bennett, The index of Leading Cultural Indicators (New York: Broadway Books, 1999), p.230 16

19. Two Time Points are better... Alternative Sort Murder Rates in Ten Largest US Cities, 1995-1998 19951998Net Change Los Angeles24.511.8- 12.7 New York16.1 8.6- 7.5 San Antonio14.2 8.1- 6.1 Philadelphia28.223.3- 4.9 Detroit47.643.0- 4.6 Phoenix19.715.1- 4.6 Chicago30.025.6- 4.4 San Diego 7.9 3.5- 4.4 Houston18.214.1- 4.1 Dallas26.523.1- 3.3 * Murder and non-negligent manslaughter per 100,000 population Source: Statistical Abstract 2000 CD-ROM tables 332; and Bureau of Justice Statistics: http://www.ojp.usdoj.gov/bjs/data/cities92.wk1 17

20. Time Goes Left to Right? National Totals ’10 % ’09 % ’08 % ’07 % ’06 % ‘05 % ’04 % ’03 % Favor 4634394144 3633 Oppose 5262565550526165 Don’t Know 2 4 5 4 6 4 3 2 Do you favor or oppose allowing students and parents to choose a private school to attend at public expense? 18 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

21. Time Goes Left to Right! National Totals ’03 % ’04 % ’05 % ’06 % ’07 % ‘08 % ’09 % ’10 % Favor 333644 41393446 Oppose 6561525055566252 Don’t Know 2 3 4 6 4 5 4 2 Do you favor or oppose allowing students and parents to choose a private school to attend at public expense? 18 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

22. 19 Storage G. Oehlert, rev. by S. Weisberg, School of Statistics, University of Minnesota, 2/2008

24. A Few Last Things To Remember  Align your data.  Avoid Fancy Fonts (use bold for titles and headings).  Arrange numbers in columns for comparison (so they are closer and the digits line up).  Use row or column summaries (e.g., means or totals), to provide a standard of usual.  Remove excess lines/boxing -- thin, straight, borders under the title and heading cells and under the main body of data.  Use space to emphasize groups/gaps; use caution though, excess space breaks adjacency.  Power point tables should usually have fewer than 24 data points. G. Oehlert, rev. by S. Weisberg, School of Statistics, University of Minnesota, 2/2008 21 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

25. Example Table 1.3: Participation and Accomplishments for Students in College Access Programs, 2009 City 1City 2Total PROGRAM PARTICIPATION Percent of Students/Families who were involved in: N=239N=334 N=573 Mentoring44%53%49% SAT/ACT Prep54%41%47% Accelerated Coursework2%6%4% Parent Education51%46%49% Financial Aid Information 84%87% 86% PARTICIPANT MILESTONES Percent of Students who: N=61N=79 N=140 Completed FAFSA*75%84%80% Submitted College Applications*68%87%78% Reported College Acceptance* [Applicants Only] 72%79%76% *These data are only for grade/age eligible students. 22 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

26. Example Table 3: Changes in Average Attendance at New and Mature Sites, 2007-08 – 2008-09 New SitesC HANGE Mature SitesC HANGE Avg. Total Hours 2007-08 78.6 +86% 132.4 +26% Avg. Total Hours 2008-09 146.5  166.9  23 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

27. Spring 08Spring 09 ASP PROGRAM CONNECTIONS Attended prior year 45%66%  Attended prior summer 32%51%  Plans to come next summer 42%55%  Plans to come next school year 45%70%  Example Table 6: Self-Reported Attendance and Plans to Continue Attending Afterschool Programs, Spring 08 v. Spring 09 24 Bruner Foundation Rochester, New Yor k Anita M. Baker, Evaluation Services

28. BYA Initiative: Key Finding #1 Substantial and Increased Enrollment in Beacon Programs NYC New Sites NYC Mature Sites SF New Sites Enrolled 2008-09 (YR 2)1140  915  Enrolled 2007-08 (YR 1) 964 817 Enrolled 2009-10 (YR 2)2211  Enrolled 2008-09 (YR 1) 1846 Table 5a: Two-year Enrollment Comparisons for Selected Beacon Programs, NYC and SF* *Unit record data made available by Beacons-on-Line and CMS databases 25 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

29. Spring 09 N=232 Activities were interesting 91% I got to learn new things 91% There were things going on that made me want to stay involved 87% Activities were challenging 83% Youth were encouraged to participate by staff 91% Youth were encouraged to participate by other youth 86% Table 6b: Percent of Cohort Survey Respondents Who Reported the Following Efforts to Engage and Retain Them Happened Regularly at their Beacons 26 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

30. Spring 08Spring 09 220232 OVERALL RATING Excellent35%42% Very Good15%19% Good13%17% Okay18%14% COMPARISON TO LAST YEAR* A Little Better this Year46% A Lot Better This Year38% 81% 92% * Answered only by those who had been at the Agency during 2007-08. Table21: Percent of Survey Respondents Who Provided Positive Feedback to Agency, Spring 08 v. Spring 09 84% 27 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

31. Percent of Training Participants (N=93) who Think AAV Training Will Help Them:  SomeA LotTOTAL Discuss issues of violence with clients44%56%100% Access additional strategies for self-care/stress reduction47%51% 98% Provide positive interventions for clients32%65% 97% Understand the importance of self-care/stress reduction58%38% 96% Offer clients new ways to: De-escalate Situations31%67%98% Manage Anger54%43%97% Do safety planning45%52%97% Conduct Bystander Interventions39%58%97% Table 3: Percent of Participants Who Think They Will Be Helped by Allied Against Violence (AAV) Training Note: the difference between the total and 100% is the proportion who indicated they did not expect the training would help them at all. For each question fewer than 5 participants indicated the training did not help them at all. 28 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

32. 29 Graphs, Graphs, Graphs, Graphs, Graphs, Graphs, Graphs Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

33. Thinking About Tables and Figures Figures, which include graphs/charts and pictures or any other visual display also must have a figure number and title (be consistent). Like tables, use the title to describe what is really in the figure. Figure 1.3 Exit Status of 2006 Domestic Violence Program Participants. For bar and line graphs, both the X  and Y  axes must be clearly labeled. The legend, clarifies what is shown on the graph. You can also add individual data labels if needed. For any bar or line graph with multiple data groups, be sure to use contrasting colors – that are printable in black and white. 30 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

34. Graph Basics  Know and understand your data - PLUS you need a good sense of how the reader will visualize the graph.  Beware poor choices or deliberately deceptive choices that provide a distorted picture of numbers and relationships.  Minimize or eliminate any element that does not aid in conveying what the numbers mean. 31 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

36. A completely irrelevant map of the world. Two entirely different kinds of 3-D charts displayed at two different perspectives. Country names are repeated three times. To display 24 numeric data points, 28 numbers are used to define the scales. The countries are sorted in no apparent order (not even alphabetically). Note the use of the letter " I " to separate the countries on the bottom chart. 33 Chart I: Extraneous Features Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

37. 34 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

38. Wall Street Journal Data Distortion http://lilt.ilstu.edu/jpda/charts/bad_charts1.htm#Junk 35 Wall Street Journal editorialWall Street Journal editorial, "No Politician Left Behind, Lack of money isn't the problem with education." Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

39. 36 The data on spending is not adjusted for inflation or, the growth in the number of pupils. In theory, 500 is the maximum score on the NAEP scale-scored math tests, but no student ever reaches this standard. * The average score for high school seniors on the same scale is just over 300. Data Distortion Examples Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

40. 37 Including more recent data, and adjusting the reading score scale, we get a much different picture: more recent data Data Distortion Adjustments http://lilt.ilstu.edu/jpda/charts/bad_charts1.htm#Junk Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

41. More Graph Basics  There are 3 main types of graphs.  Pie Charts (composition)  Bar Graphs (description, comparison)  Line Graphs (trends over time)  There are 3 parts to a graph: labels, scales, graphical elements.  Label - titles, axes, legends, data series  Scales especially for Y  Graphical elements (e.g., the bar in the bar graph). 38 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

42. Figure 1: Mean Tuition & Fees, Per Semester Illinois Public Universities, 2001 University Title Legend Grid Line X-axis label X-axis title Y-axis scale $4340 $3710 $3387 Data Label Graphical Elements 39 Bruner Foundation Rochester, New

43. Graph Specifics  Title defines what is in the graph, or states the conclusion you want the reader to reach.  Axis Titles should be brief -- do not use if the info is clear from the table title.  Axis Scale/Data Labels – define magnitude:  Avoid using too many numbers  If you label the value of each individual data point do not label the Y axis. If it seems necessary to label every value in a graph – consider using a table. 40 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

44. Graph Specifics - Continued  Legends required for multiple data series.  Inside or on the bottom is best location (NOT outside)  Gridlines – if used at all should use as little ink as possible.  The amount of ink given to non-data elements should be limited.  Plot area borders or shading are unnecessary.  AVOID using any un-necessary 3-D effects. Keep graphs simple, but don’t underestimate your reader. If it’s better said than shown, then say it. 41 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

45. Rules for Pie Charts  Avoid using pie charts  Use pie charts only for data that add up to some meaningful total.  Never use three-dimensional pie charts  Avoid forcing comparisons across more than one pie chart. http://lilt.ilstu.edu/jpda/charts/bad_charts1.htm#Junk 42 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

46. Pie Charts Show Composition of a Whole Group 43 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

47. Rules for Bar Charts  Minimize the ink. Do not use 3-D effects.  Sort the data on the most significant variable.  Use rotated bar charts (i.e., horizontal) if there are more than 8 – 10 categories  Place legends inside or below the plot area  Keep the gridlines faint.  With more than one data series beware of scaling distortions. Bar charts often contain little data, a lot of ink and rarely reveal ideas that cannot be presented more simply in a table. 44 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

48. Y- Axis Likert Scales and Bar Charts – A Definite No-No Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

49. Graphs are Supposed to Inform, not Entertain Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

50. Bar Graphs Show Frequencies Horizontal or Vertical 47 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

51. Percent that persisted Bar Graphs Show Frequencies Horizontal or Vertical 48 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

52. Bar Graphs Show Frequencies Horizontal or Vertical 49 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

53. 50 Bars Can be “Clustered” to Show Differences or Change Percent of HS Graduates Enrolled Figure 3b College Enrollment Among Students from Different Types of High Schools, 2003-2009 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 70

54. 51 Bars Can be “Clustered” to Show Differences or Change Percent of Students With College Enrollment who Graduated College Figure 4b Proportion of HS Graduates with College Enrollment Who Graduated College, 2003 v. 2005 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

55. Percent of CSI Participants with High Attendance (100 or more hours), by Year Bar Graphs Can: Show Change Over Time, Show Targets, Be Enhanced 52 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services Target

56. Stacked Bars Show Distributions Figure 12 Types of Colleges First Attended for HS Graduates 2003-2009 Public Private Four Year Two Year In State Other Four College Attributes 53 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

57. Stacked Bar Charts: Advice  Use with caution especially when there is no implicit order to the categories.  Stacked bar charts work best when the primary comparisons are to be made across the data series represented at the bottom of the bar. 54 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

58. Figure 3: Survey Results: Percent of Principals Who are Satisfied with K – 3 Literacy Achievement at Project Schools and Comparison Schools 66% 55 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

59. 56 Line Graphs Show Change Over Time Figure 6.7 Proportion of Students Passing Proficiency Test TEST YEAR Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

60. Percent of Program Completers Who Implement Suggestions, Still Need Help, Over Time N=1252

61. Rules for Time Series (Line) Graphs Time is almost always displayed on the X- axis from left to right. Display as much data with as little ink as possible Make sure the reader can clearly distinguish the lines for separate data series Beware of scaling effects When displaying fiscal or monetary data over-time, it is usually best to use deflated data (e.g., inflation-adjusted). 58 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

62. 59 K Student Outcomes (2008-09) Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

63. FCPS Selected Findings  While the numbers of plans have increased, time to develop them has substantially decreased. DAYS TO COMPLETE PLAN Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

64. Figure 1: Percent of Respondents Who Selected Each of the Following as one of the Top Three Reasons They Subscribe to the Forum. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 61

65. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 62

66. Figure 2: Daily cost of Incarceration in Connecticut: Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 63

67. 64 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

68. 65 Figure 1: Presence of DCF-Involved Children at Hartford and Manchester TOP Sites Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

69. 68 Location of M.O.B. Participants, Falls Prevention Program Year 1 (2013-14) 66

70. 67

71. Use items from participants that you photograph. Be sure to protect confidentiality Use items from 68

72. CONCLUSIONS  Marshall Fund resources have been used to provide important programs in multiple areas.  Careful attention has been paid to donor intent.  Programs and services have been promoted around the county. Distribution is not completely equitable, but there is widespread use.  Services have contributed to desired outcomes. 69 Use stock photos

73. Consider infographics! Graphic visual representations of information (www.coolinfographics.com. 70

75. 72 Math for Evaluators The Magic of Proportions 1)To calculate percentage divide the numerator into the denominator. Think Notre Dame (N/D) – but the denominator must be specified carefully. Total percents include all possible. Valid percents include only those for whom there is data. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

76. 73 Math for Evaluators The Magic of Proportions FrequencyTotal PercentValid Percent No, Not Really 6229.733.5 Yes, Somewhat 6631.635.7 Yes, Definitely 5727.330.8 Total 18588.6100.0 System Missing2411.4 TOTAL209100.0 Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

77. 74 Math for Evaluators The Magic of Proportions 2)You can combine percentages for the parts of a whole group by adding. 3)Response “rate” is a special kind of percentage: *Denominator must be adjusted for non-viable administration (e.g., returned mail) *Desirable response rates (like targets) must be determined in advance. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

78. 75 Math for Evaluators Showing Change 1) To calculate percent change: (NEW VALUE – OLD VALUE)/OLD VALUE * To check this multiply: old value * 1.change value Use this when you are comparing two numbers. Label with units. 2) To calculate percentage point change: (Time 1% - Time 2%) OR (Time 2% - Time 1%) Label as percentage point change Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services

79. 76 Math for Evaluators Showing Change: Examples 20062007Difference Total Participants174190 Plans Completed125167 % Completing Plans 72%88% Do the math: Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services