Intro to NRS Data Diving Mary A. Gaston, Ed.D. & Jennifer Cooper-Keels February 4, 2011

Why Look at Data? Data help us to… Replace hunches and anecdotes with facts concerning the changes that are needed; Identify root causes of problems; Identify whether student or program goals are being met; and Tell our stakeholders, including students, about the value of our programs and the return on their investments. Ask participants the question—let them suggest answers—then show what we have, tying into what they have said. 11/15/2018

Data: A Carrot or a Stick? Data may be used… To highlight, clarify, and explain what’s happening in your program OR To show what’s not happening in your program. “However beautiful the strategy, you should occasionally look at the results.” –W. Churchill 11/15/2018

Data Tell You Where you’ve been Where you are Where you’re going How to get there Data can help you design a quality program to help meet learners’ goals. 11/15/2018

Applied to Adult Education… What can data do? Guide you to improve instruction Measure program success & effectiveness Tell you if what you are doing is making a difference Tell you which classes are getting the results you want—and which are not Get to the root of problems, such as poor retention, low educational gains, or low transition rates Let participants answer the question, “What can data do?” and list on chart paper—then go over the suggestions on the slide. 11/15/2018

Starting the Dive

Starting the Dive Educational Gain While there are a number of measures in our Data System, the one we will focus on today is: We really have three main measures in our data system. These are (let them name): Attendance, Educational Gain, and Transition Outcomes (goals). For this workshop, we are going to focus on Attendance and Educational Gain. Educational Gain 11/15/2018

Educational Gain Advancement through 12 educational functioning levels Core NRS measure Can tell us: Whether the program/students are meeting goals Which sites/classes/teachers are most effective Extent of student progress Impact of changes 11/15/2018

Examples: What Increases Ed Gain Make classes learner-centered Focus on relevant knowledge Opportunity for practice and application Coherence Sufficient Intensity and Duration  (NRC, 1999; Garet, Porter, Desimone, Birman, & Yoon, 2001) Ask participants to brain-storm about what increases educational gain . . . Have them report out and make a list. Then—compare with what we have. 11/15/2018

Data analysis is only as good as the original data allow. Do You Trust Your Data? Data analysis is only as good as the original data allow. Keys to good data collection systems include: Clear policies and procedures for data entry Data is entered & reviewed daily, weekly, or monthly Teachers, staff, administrator all have access to data and review regularly Teachers share data with students Let participants brainstorm about what their own programs do to improve data quality as well as data access. Let some report out. What does your program do to ensure data is accurate, reporting is timely, and staff have access to the data? 11/15/2018

Take a Dip in the Data Pool

Dive into the Data Pool For each of the next few slides write down your observations for discussion What do you see? What is interesting or unusual? Do any questions or hypotheses come to mind as a result? 11/15/2018

Write Observations/Questions? Make sure participants understand what they are seeing here. Give participants time to study the chart and write down observations. Then, in small groups, have them share observations and come up with ideas for where to go next? What other data do they want to see? 100 11/15/2018

Write Observations/Questions? Make sure participants understand what they are seeing here. Give participants time to study the chart and write down observations. Then, in small groups, have them share observations and come up with ideas for where to go next? What other data do they want to see? 11/15/2018

Write Observations/Questions? For time’s sake, we are limiting the dive today to 4 GED classes, represented on this chart. Make observations and come up with hypotheses about what might be happening in these classes. What other info do you want? 11/15/2018

Write Observations/Questions? Here are overall completion rates (educational gain) as a comparison—included is the statewide average for perspective. Have participants make observations—and again, come up with other data they might want to see. 11/15/2018

Write Observations/Questions? For time’s sake, we are limiting the dive today to 4 GED classes, represented on this chart. Make observations and come up with hypotheses about what might be happening in these classes. What other info do you want? 11/15/2018

Write Observations/Questions? Main AM This chart compares the 4 classes’ retention rate (avg. attendance per student) with respect to the number of class hours per week. Observations? Other questions/data you want to see? Young Adult Main PM Satellite R 11/15/2018

Write Observations/Questions? Three variables here—size of bubble represents number of students served in the class that year. Each bubble is plotted based on educational gain and retention rate. Observations? Questions? Suggestions for change? 11/15/2018

Where to Go From Here? Based on what was learned from this “data dive”: What should I change or replicate? What data supports this change? What additional data should be reviewed? What is the timeframe for change? Is it realistic? What obstacles/barriers will we encounter? What is the follow-up plan to measure and evaluate change? 11/15/2018

You Just Did a Data Dive! 11/15/2018 Review what has just happened—how we have walked through an NRS Data Dive. Share with participants that the Excel spreadsheet online has several data dives already worked out—BUT— DO NOT limit yourself to just these dives—you need to have a system but answer the questions you need answered with the data/info you have. 11/15/2018

Questions: Educational Gain Sample question Further questions Data collection & quality What is the range of pre/posttest scores in my program/site? Are all the test scores within the correct range for the test and class level? Students Which students are most likely to complete a level (student characteristics)? Do students with higher contact hours have greater completion rates? Teachers What teacher characteristics are most related to level completion? How high is teacher turnover at each site? Which sites retain teachers longest/best? Instruction Which instructional approaches have the greatest impact on gain? Do assessments match course content? Program How many hours of PD do our teachers participate in? Which PD have the greatest impact on student learning? Program policy Do placement policies differ among sites? Which placement policies have an impact on educational gains?

Tools for Data Diving 11/15/2018 Time left? Let programs use Handout Five in packet to begin planning their own data dive. 11/15/2018

What questions do you want to answer about your own local program? What Do I Want to Know? What questions do you want to answer about your own local program? Time left? Let programs use Handout Five in packet to begin planning their own data dive. 11/15/2018