1. Damian Bebell James Burraston Center for the Study of Testing, Evaluation, & Educational Policy Lynch School of Education Boston CollegeDec 15, 2014 Natick Public Schools 21 st Century Teaching and Learning Study: Evaluation Results

2. Background and perspective Empirical educational research, measurement, and evaluation Background in psychometrics, measurement, and evaluation Leveraging technology to advance the state of the art in educational research and measurement… Data for the sake of data vs Ownership of “success” and ongoing formative use of information

3. Quantifying computer access US Average Student : Computer Ratios 1983 = 125 : 1 1998 = 6 : 1 2007= 4 : 1 2010=3 : 1 Source: Market Data Retrieval, Public School Technology Survey 1998-2006; Education Week, 2007; Gray, Thomas, & Lewis, 2010

4. Shared access Sporadic use Limited impact 1:1 Computing The rise of 1:1 Computing 1:1 computing programs continuously evolving Widespread proliferation across world Few programs have more than 5 years experience Fewer still have invested in empirical research and evaluation efforts So, lots of interest but limited research and evidence

5. 1:1 student computing research 1:1 defines critical technology access…pedagogical inferences Emphasis on selection and mechanics of program, more limited understanding of how teaching and learning change Building level leadership Pedagogical and content support for teachers Systematic reflection, asking questions, and willingness to adapt/evolve Many different definitions of success

6. Natick/Boston College Research Partnership September 2012- June 2014 Design and conduct external study to document impacts and changes as students and teachers transition to new technology rich computing environments. Document through a series of reports and presentations potentially evolving teaching and learning practices afforded by the rich learning settings. Document how Natick teachers and students accessing, using, and relating to technology in support of teaching and learning. Provide NPS with a rich empirical model (and data) for all future inquiries and investigations concerning the relationship between technology practices and a variety of outcomes. www.bc.edu/natick21

7. Natick/Boston College Research Partnership Acknowledgments School Leadership Teachers Students District Leadership Natick Community www.bc.edu/natick21

8. Long Term Goals/Outcomes Student Outcomes/Program Goals Decrease behavioral and attendance issues Decrease special education/ACT referrals Increase in students’ computer use time in all subject areas Increase in student use of computers for doing homework in all subject areas Increase in homework completion Staff/Teacher Outcomes/Program Goals Increase in teachers use of technology-based resources to enhance instruction Increase in teachers allowing students to use multiple technology-based resources to address assessment demands Increase in the frequency that teachers structure learning opportunities that: are inquiry-based, include a writing component, allow students to engage with and utilize technology-based resources, and allow students to demonstrate mastery In Year 1 (2012/2013), revision of at least one unit per trimester where technology is used to transform learning from a traditional teacher-directed curriculum-based activity to a blended learning experience with an inquiry-based mode of facilitating learning Decrease the number of teacher complaints concerning the lack of technology resources available for instruction

9. NHS Research/Evaluation Design 7 th -12 th Student Survey (*Fall 2010, Fall 2012, Spring 2013) *92% of the NPS seventh grade student population participated in a pilot survey in Fall 2010. 7 th -12 th Teacher Survey (Fall 2012, Spring 2013) Classroom Observations Focus Groups Staff Interviews Secondary analyses of school records (achievement, attendance, Special Ed status, course patterns) Student Video Competition

10. NHS Student Video Competition “How have computers and technology changed the way you learn new things and/or how your teachers teach.” www.bc.edu/natick21

11. Survey Response Rates/Parent Consent

12. So, what did we learn?

13. Selected Results Intro to Results Dramatic increase in student tech use/shift in teacher practices Exploring trends within the student results Student and Teacher Beliefs Exploring Quantitative Impacts and Outcomes

14. Results: Students’ increased use of technology across grades and subject areas

15. Major increase in students’ frequency of technology use across core classes (2010-2012):

16. Major shifts in students’ classroom practices (2010-2012)

17. Major shift in teacher’s frequency of technology use in the classroom (2010-2012):

18. Shifts in teachers’ classroom computer use according to students (2012-2014)

19. Where (and how often were) NHS students using technology in 2014?

20. Digging deeper in the Results: Exploring trends within student practices (2013-2014)

21. Exploring gender differences in frequency of NHS boys and girls technology use across their classes

22. Exploring differences in student and teacher technology use across NHS students enrolled in one or more honors courses

23. Exploring differences in student practices comparing NHS students enrolled in one or more honors courses

24. Exploring differences in student practices across socio-economic status for NHS students

25. Results: Natick Student and Teacher Attitudes and Beliefs

26. Natick students report using a computer in school improves their education:

27. Natick teachers report using a computer in school improves their teaching:

28. Results: Empirically examining student outcomes associated with 1:1 Computing and new facilities

29. Impacts on NHS student homework completion Average number of minutes completing home per night and the percent of homework completed using a computer

30. Impacts on NHS student homework completion Average homework completion rates self-reported by students

31. Impacts on NHS Student Attendance Average student absences before and during first year 1:1 implementation

32. Impacts on NHS Student Attendance Distribution of student absences before and during the first year of 1:1 computing There appears to no relationship between students reported frequency of technology use in school and their absences

33. Impacts on NHS Student Discipline Paired-sample t-tests examining change in frequency of detentions from the 2011-2012 school year to 2012-2013.

34. Impacts on NHS Student Discipline Independent-samples t-tests comparing students technology-related disciplinary problems from 2012/2013 to 2013/2014 across grade levels

35. Impacts on NHS Student Discipline Average teacher and student perceptions of students' ability to get around the school Internet filter

36. Impacts on NHS Special Education Percentage of students who received various levels of special education services during the last year in the old high school building and the first year in the new building.

37. Impacts on NHS student achievement Average ELA and Math MCAS scaled scores over time for the NHS class of 2015 and 2016

38. Impacts on NHS student achievement Percentage of students who scored "proficient" or better on the MCAS from 6th to 10th grade for classes of 2015 and 2016

39. Impacts on NHS student achievement Percentage of students who scored as “proficient” or higher on the 10th grade MCAS from 2009 to 2014.

40. Revisiting Long Term Goals/Outcomes Student Outcomes/Program Goals Decreasing behavioral and attendance issues Decreasing special education/ACT referrals Increase in computer use time in all subject areas Increase in student use of computers for doing homework in all subject areas Increase in homework return rates Staff/Teacher Outcomes/Program Goals Increase in the frequency that teachers use technology-based resources to enhance instruction Increase in the frequency that teachers allow students to use multiple technology- based resources to address assessment demands Increase in the frequency that teachers structure learning opportunities that: are inquiry-based, include a writing component, allow students to engage with and utilize technology-based resources, and allow students to demonstrate mastery In Year 1 (2012/2013), revision of at least one unit per trimester where technology is used to transform learning from a traditional teacher-directed curriculum-based activity to a blended learning experience with an inquiry-based mode of facilitating learning Decrease the number of teacher complaints concerning the lack of technology resources available for instruction

41. Questions/Follow Up Damian Bebell, Principal Investigator bebell@bc.edu James Burraston, Research Associate james.burraston.1@bc.edu www.bc.edu/natick21