Making the Most of Your Data: Strategies for Evaluating Your Program Greta Colombi, NDTAC; and John McLaughlin, ED

2 How Do You Use Your Title I, Part D, Data? Do you:  Look at your Consolidated State Performance Reports (CSPRs) each year?  Incorporate what you know into applications for funding?  Make statements about how these tables do not really reflect what goes on in the classroom?  Attempt to explain them to your stakeholders?  Put results away in your files?  Hope that the CSPR will go away before next year?

 Why using data is important  How data can be used  Strategies for using data  Activities 3 Agenda

 Consider ways you can better use your data  Consider what technical assistance (TA) and support you could provide to your subgrantees to encourage data use 4 Outcomes

5 Data Tells You and Your Programs...  Where you’ve been  Where you are  Where you’re going  How to get there

6 Barriers to Using Data  Your program’s data are handled separately from your program.  Your program’s culture does not focus on data.  Gathering data is perceived to be a waste of time.  Staff lack adequate orientation and training in the value of data collection.  Staff have had negative experiences with data collection.  Staff are not aware of other programs’ successes in using data.  Staff think that data are collected “just for the State or the Feds.”

7 Working With What You Have The same data you collect and report…  Demographics of students (race/ethnicity, age, and gender)  Academic performance in reading and mathematics  Academic and vocational outcomes  Student and facility counts  Program spending* can be used for…  Accountability  Program promotion/marketing  Program management and improvement * States do not report program spending within the CSPR, but should have this information at hand.

8 Functions of Data  Help us identify whether goals are being met (accountability)  Tell our departments, delegates, and communities about the value of our programs and the return on their investments (marketing)  Help us replace hunches and hypotheses with facts concerning the changes that are needed (program management and improvement)  Help us identify root causes of problems (program management and improvement)

9 Program Components by Data Function Program Accountability Program Marketing/ Promotion Program Improvement Student demographics Are the appropriate students being served? How are you addressing the needs of diverse learners? Which students need to be better served? Student achievement Are students learning? What are students learning? What gains have they made? How can we help improve student achievement? Student academic outcomes Are students continuing their education? What are students doing to continue their education? How can we help improve student academic outcomes?

10 Strategies for Improving Data Use  Accountability –Monitor data based on national benchmarks –Set State benchmarks and monitor program performance  Program Improvement –Evaluate program (formative and/or summative)  Marketing –Develop and distribute State/program report cards

11 Data Use Improvement Activities  Meet with SEA staff (data, programmatic) to analyze the data you have  Request disaggregated data from subgrantees/ programs to improve data use  Communicate findings with subgrantees  Support subgrantee/program evaluations –Communicate allowability of funding –Include evaluation requirements in program applications/formal agreements –Provide TA at conferences/meetings or during monitoring on benefits of using data and how to do so

12 Data Use Model Two Components of the Model:  Data Analysis and  Program Improvement Model developed by the National Reporting System for Adult Education support project at the American Institutes for Research

13

14 Focusing the Question Break the question into inputs and outcomes:  Inputs (what your program contributes):  Teacher education, experience, full-time/part-time  Instructional curriculum  Hours of instruction per week  Outcomes (indicators of results):  Improved posttest scores  Completed high school  Earned GED credentials

15 Focusing/Refining the Question (1) Poor Question:  Does my program have good teachers? Good Question:  Does student learning differ by teacher? Better Question:  Do students in classes taught by instructors who have more teaching experience have higher test scores than those taught by new teachers?

16 Focusing/Refining the Question (2) Poor Question:  Is my program helping the most needy students? Good Question:  Are students who are below grade level learning less in my program than other students? Better Question:  Are students who are below grade level advancing levels at the same rate as students at grade level?

17 Developing a Data Analysis Plan  What data do you already have that will answer your question?  What additional data, if any, will you need to answer your question?  If so, where will you get the additional data?  What’s your plan for obtaining the data you need— and what’s your timeline?

18 Analyzing and Interpreting Your Data  Keep your original question in mind.  Look for patterns and differences.  Use appropriate data and statistics.  Disaggregate the data.  Consider data quality.  Draw appropriate conclusion(s).  Remember serendipity: Be open to the unexpected.

19 Presenting Your Data (1) Frequency Tables  Show numbers and percentages by category, e.g., ethnicity, gender, age.  Provide crosstabulations, e.g., ethnicity by age.

20 Presenting Your Data (2) Graphs and Charts Bar Chart:  Categories are displayed as bars, e.g., students by age. Pie Chart:  A slice of the pie shows proportion of the whole, e.g., various ethnicities of total students. Line Chart:  Data form a continuous measure/trend (not categories), e.g., posttest scores.

21 Presenting Your Data (3) Communication Strategies  Article by education reporter in local newspaper  Public meeting or news conference presented by superintendent or dean  Newsletters  Special events, e.g., open house  Web sites  Annual report

22 Conclusion  Using your data can help you (1) ensure accountability, (2) make program improvements, and (3) market your program.  It is key to look at your data, involve others, and consider how you can use the data.  You can integrate data use activities in regularly scheduled activities.

23 Activities  Activity 1 –Discuss possible reasons for the scenarios included in the handout –Consider: (1) What additional data would you need to better understand the root cause of the problem? and (2) What could be done about it?

24 Activities  Activity 2 –Review the scenario –Consider how you could address the issue(s) in regularly scheduled activities