Measuring the Effects of Collaboration and Professional Development on the Technology Integration and Student Achievement in K-12 Classrooms Melinda J. Mollette and Jessica D. Huff American Evaluation Association conference November 2009

Why Implement a School Technology Program? Based on the idea that effective school library media and instructional technology programs support both effective teaching and learning “Technology is generally not a direct cause of change but rather a facilitator or amplifier of various educational practices” - Lesgold, 2003 School library media and instructional technology programs are key to making education relevant. Lesgold, A. (2003). Detecting technology’s effects in complex school environments. In Evaluating Educational Technology: Effective Research Designs for Improving Learning (Means, B., and Haertel, G., Eds.). New York: Teachers College Press.

IMPACT Components Structured Collaboration Supportive Leadership Technology Integration Flexible Access to Media Ctr/Labs Professional Development Resources/Equipment Key Personnel (e.g. Media Coord, Instr’l Technology Facilitator

IMPACT Cohorts IMPACT I Elementary/2 Middle Schools One School in each LEA IMPACT II All Middle Schools One School in each LEA IMPACTing Leadership Quarterly Prof’l Development Workshops – 3 days each (2006/07) Follow-up support – 2007/08 Funding distributed July 2008 IMPACT III IMPACTing Leadership districts/schools District-wide initiative – gr. K-12 in two and K-8 in one LEA IMPACT IV Weeklong summer professional development workshops for school-level MTAC members Central office personnel attended IMPACT Academy District-wide initiative at upper grade levels (only gr. 3-12) in four LEA’s

Results from IMPACT III/IV teacher surveys 85% of teachers in IMPACT schools feel their principal is frequently/always committed to providing teachers with opportunities to improve instruction (SAI – April ‘09) 89% of teachers report they frequently/always have opportunities to learn how to use technology to enhance instruction (SAI-Apr09) 53% of teachers report they frequently/always set aside time to collaborate about what they learned from their PD experiences (SAI – April ‘09) Teachers’ self-reported technology skills (NETS-T) improved significantly from Fall 08 to Spring 09 (p<.000) in IMPACT IV schools

Results from IMPACT III/IV – SAI (Spring ‘09) 37% of teachers reported they frequently/always observe each other’s classroom instruction as one way to improve teaching 54% reported they frequently/always received feedback from colleagues about classroom practices. 79% said teacher learning was frequently/always supported through a combination of strategies (e.g. workshops, peer coaching, study groups, joint planning of lessons, and examination of student work). 67% reported they frequently/always receive support implementing new skills until they become a natural part of instruction. 49% said they frequently/always get to choose the kind of prof’l development they receive

NETS-T Survey - % in highest response category across all 48 items combined IMPACT Model Spring 2008Fall 2008Spring 2009 Overall Change IMPACT III-all levels19.5%18.2%18.4% -1.1% Elementary17.3%15.8%15.3% -2% Middle21.9%22.4%20.6% -1.3% High22.7%18%20.1% -2.6% IMPACT IV – all levels19.3%25.6% +6.3% ElementaryN/A19.6%23.4% +3.8% MiddleN/A21.5%27.1% +5.6% HighN/A17.6%25.8% +8.2% Response categories: 3-Able to Teach Others; 2-Confidently (knowledgeable and fluent); 1-Minimally (need help); 0-Not at all

IMPACT III/IV focus groups Having a full-time instructional technology facilitator on staff at each school was a crucial factor enabling teachers to access (and use) a broad array of instructional strategies and resources. Training needs to accommodate a variety of skill levels, from beginners, to more advanced users. Essentially, provide “differentiated instruction” for the teachers, as well as the students Implementing the IMPACT Model district-wide provided reinforcement, enthusiasm and support from central office as well as parents and the community. However, in some cases, decisions were made “at the top” with less input.

Results from IMPACT III – 2007 to 2009 NC End of grade tests in Math (Gr.3-8) IMPACT students were 42% more likely than comparison group to increase achievement levels 46% more likely than comparison group to improve from not passing to passing (p<.000) Economically disadv’d students were 54% more likely than ED students in comparison schools to improve from not passing to passing (p<.000) NC End of grade tests in reading (Gr. 3-8) IMPACT students were 22% more likely than comparison group to improve from not passing to passing, from 2008 to 2009 (p=.020) Largest avg change in Rdg scores from 2007 to 2009 was among IMPACT middle school students

Average Reading Scale scores by school level * HLM analysis indicated a significant 3-way interaction (time x school level x school enrollment): Ƴ 13 =.8539, t= 4.18, p<.0001

Changes in Math scores – IMPACT III cohort

Odds of passing and scoring above grade level In 2007, IMPACT III and comparison students in gr. 3-8 were equally likely to pass Math (p=.115) In 2009, IMPACT III students were 12% more likely to pass than comparison students. (p<.000) HLM analyses showed a significant school enrollment x year interaction, such that the change/growth over time in Math scale score was dependent on the students’ project school enrollment. ( Ƴ 11 = , t= 4.72, p<.0001) Students enrolled in IMPACT schools showed slightly stronger growth (avg. increase=5.2 points) in Math scores than students enrolled in comparison schools (avg. increase=3.85 pts).

Results from IMPACT IV – 2008 to 2009 MATH – Gr. 3-8 IMPACT IV students were 11% more likely than comparison group to increase their Math performance level (p=.022) Economically disadvantaged students in IMPACT IV schools were 12% more likely than ED students in comparison group to increase performance levels (p=.04) READING – gr. 3-8 IMPACT IV students were significantly more likely to pass EOG-Reading in 2009 than 2008 (OR=1.133, p=.004), comparison schools did not show similar improvement (p=.204).

Future of IMPACT Most of the schools from the IMPACT III/IV cohorts, as well as the IMPACT II cohort, have received additional funding to continue IMPACT through the 2010/2011 school year. IMPACT high schools will be using addit’l funding to purchase students laptops (i.e. become 1:1 schools) IMPACT elem & middle schools will use addit’l funding to purchase/upgrade teacher laptops & other equipment

For more information IMPACT Model Document from setda.org regarding IMPACT 281&name=DLFE-432.pdf NC Dept of Public Instruction Director on Instructional Technology Neill Kimrey –