Intro to NRS Data Diving Mary A. Gaston, Ed.D. & Jennifer Cooper-Keels February 4, 2011

2 2 Why Look at Data? Data help us to… Replace hunches and anecdotes with facts concerning the changes that are needed; Identify root causes of problems; Identify whether student or program goals are being met; and Tell our stakeholders, including students, about the value of our programs and the return on their investments. 5/24/2015

3 3 Data: A Carrot or a Stick? Data may be used… To highlight, clarify, and explain what’s happening in your program OR To show what’s not happening in your program. “However beautiful the strategy, you should occasionally look at the results.” –W. Churchill 5/24/2015

4 4 Data Tell You Where you’ve been Where you are Where you’re going How to get there Data can help you design a quality program to help meet learners’ goals. 5/24/2015

5 5 Applied to Adult Education… What can data do? Guide you to improve instruction Measure program success & effectiveness Tell you if what you are doing is making a difference Tell you which classes are getting the results you want—and which are not Get to the root of problems, such as poor retention, low educational gains, or low transition rates 5/24/2015

Starting the Dive

7 7 Attendance Educational Gain Transition Outcomes (Goals) 5/24/2015 For this workshop, we will focus on “Attendance” and “Educational Gain.” 3 Main Measures in our Data System:

8 8 Attendance Contact hours of instruction the learner receives (NRS) Includes intensity and duration Can help to tell us whether: –Instruction is successful –Content and materials are relevant –Students are motivated –Students are reaching their goals 5/24/2015

9 9 Examples: What Increases Attendance Quality instruction and relevant content Well-trained teachers Clear goals set at intake, revisited regularly, and matched to teachers and content Reduction of obstacles – flexibility in programming, support services, and access to site off-hours (NCREL; Lieb, 1991; Comings, 2007; Beder, 1988; Beder, 1991; Comings, Parella, & Soricone, 1999; Kerka, 2005; Thoms, 2001; Porter, Cuban & Comings, 2005) 5/24/2015

10 Educational Gain Advancement through 12 educational functioning levels Core NRS measure Can tell us: –Whether the program/students are meeting goals –Which sites/classes/teachers are most effective –Extent of student progress –Impact of changes 5/24/2015

11 Examples: What Increases Ed Gain Make classes learner-centered Focus on relevant knowledge Opportunity for practice and application Coherence Sufficient Intensity and Duration (NRC, 1999; Garet, Porter, Desimone, Birman, & Yoon, 2001) 5/24/2015

12 Do You Trust Your Data? Data analysis is only as good as the original data allow. Keys to good data collection systems include: Clear policies and procedures for data entry Data is entered & reviewed daily, weekly, or monthly Teachers, staff, administrator all have access to data and review regularly Teachers share data with students What does your program do to ensure data is accurate, reporting is timely, and staff have access to the data? 5/24/2015

Take a Dip in the Data Pool

14 Dive into the Data Pool For each of the next few slides write down your observations for discussion What do you see? What is interesting or unusual? Do any questions or hypotheses come to mind as a result? 5/24/2015

15 Write Observations/Questions? 5/24/

16 Write Observations/Questions? 5/24/2015

17 Write Observations/Questions? 5/24/2015

18 Write Observations/Questions? 5/24/2015

19 Write Observations/Questions? 5/24/2015

20 Write Observations/Questions? 5/24/2015 Satellite R Main PM Young Adult Main AM

21 Write Observations/Questions? 5/24/2015

22 Where to Go From Here? 5/24/2015 What should I change or replicate? What data supports this change? What additional data should be reviewed? What is the timeframe for change? Is it realistic? What obstacles/barriers will we encounter? What is the follow-up plan to measure and evaluate change? Based on what was learned from this “data dive”:

23 You Just Did a Data Dive! 5/24/2015

24 Questions: Attendance Attendance & Retention Sample questionsFurther questions Data collection & quality Who enters attendance data at each site? How often is attendance data entered? Who checks the data? How often? StudentsHow does attendance differ by student type (ESL vs. ABE)? When in the term do students tend to drop/stop-out most? Is this the same across sites? TeachersWhich classes have very high (or low) attendance? Do teachers with high attendance have greater educational gains? InstructionDoes attendance vary by instructional content (e.g. GED, workplace) or level? How many hours does it take to achieve a goal, on average? ProgramWhat is the average attendance for my program? Are my program’s attendance hours similar to other programs? Program policy Are my managed enrollment classes more successful than open classes? Does managed enrollment result in higher ed gains or greater goal achievement?

25 Questions: Educational Gain Educational GainSample questionFurther questions Data collection & quality What is the range of pre/posttest scores in my program/site? Are all the test scores within the correct range for the test and class level? StudentsWhich students are most likely to complete a level (student characteristics)? Do students with higher contact hours have greater completion rates? TeachersWhat teacher characteristics are most related to level completion? How high is teacher turnover at each site? Which sites retain teachers longest/best? InstructionWhich instructional approaches have the greatest impact on gain? Do assessments match course content? ProgramHow many hours of PD do our teachers participate in? Which PD have the greatest impact on student learning? Program policyDo placement policies differ among sites? Which placement policies have an impact on educational gains?

26 Tools for Data Diving 5/24/2015

27 What Do I Want to Know? What questions do you want to answer about your own local program? 5/24/2015