School and Teacher Effects: A Team Effort Sharon Walpole University of Delaware Michael C. McKenna University of Virginia

So how’d we do last year? Let’s look at some data from the UGA external evaluator’s report

But WHY? Think about your most effective and your least effective kindergarten teacher. They both have the same materials. They have the same professional support system. They both have the same reading block. What is it that actually differs between the two and might lead to differences in achievement?

But WHY? Think about your most effective and your least effective first-grade teacher. They both have the same materials. They have the same professional support system. They both have the same reading block. What is it that actually differs between the two and might lead to differences in achievement?

But WHY? Think about your most effective and your least effective second-grade teacher. They both have the same materials. They have the same professional support system. They both have the same reading block. What is it that actually differs between the two and might lead to differences in achievement?

But WHY? Think about your most effective and your least effective third-grade teacher. They both have the same materials. They have the same professional support system. They both have the same reading block. What is it that actually differs between the two and might lead to differences in achievement?

Individual Differences Teacher Differences School Differences Portion of variance in achievement explained Lessons from School Effectiveness Literature

An LC can form judgments about teacher differences. It’s harder to judge differences at the school level because most of us lack the opportunities to make such comparisons.

Nevertheless, factors at the school level can have strong effects on achievement. The better we understand them, the more we can control them through leadership.

Differences among teachers have a clear impact on learning. Coaching attempts to reduce some of these differences by guiding teachers toward best practice.

But what about differences among schools? Can school factors be important as well?

What are some differences among kids that affect achievement? What are some differences among teachers that affect achievement? What are some differences among schools that affect achievement?

Organize Your Hunches ChildTeacherSchool K 1 2 3

Let’s look at three studies that attempted to identify school effects on reading achievement. What can we can learn from them?

Beat-the-Odds Study* Design Features Looked at school and teacher factors Used measures of word readings, fluency, and retellings 14 schools, 11 of which “beat the odds” 2 teachers at each grade, and 4 children per classroom Relied on interviews, surveys, observations, and scores School factors positively correlated with growth Forging links with parents Using systematic assessment Fostering communication and collaboration *Taylor, Pearson, Clark, & Walpole, 2000.

State-Level Outlier Study* Design Features 2 high-achieving, 1 low-achieving from 3 clusters: country, main street, uptown Used state tests, interviews, and observations School factors present in the high-achieving schools Strong leadership and commitment Teacher knowledge Time and opportunity for children to read Commitment of 8-10 years to the change process *Mosenthal, Lipson, Torncello, Russ, & Mekkelsen, (2004).

Curriculum Effects Study* Design Features Compared 4 major reform models in first grade Each used a coach, lots of PD, and regrouping of kids 4 experienced schools in each model Measures of decoding, comprehension, vocabulary; classroom observations Comparisons All of the schools were relatively successful None of the 4 models proved best *Tivnan & Hemphill, 2005

What lessons do these studies teach?

Common Characteristics 1. No one curriculum or intervention model is a magical solution to student achievement problems 2. Intense focus on the school’s goals is associated with success Assessment, communication, collaboration Leadership, vision, knowledge PD plus specific curriculum

Differences From Us 1. These schools were successful or experienced already; we are striving to be successful and we are still new to our curricula 2. These schools had different demographics than we do (except for the Curriculum Effects Study) 3. These studies do not include the effects of intensive interventions

Limits of Generalizability 1. We can’t tell whether these characteristics are causes or characteristics of success 2. We don’t know whether these factors “transfer” to striving schools

Now let’s look at a fourth study – one with better lessons for Georgia and Reading First.

GA REA Study* Question In a two-year reform initiative, what school- level characteristics are significantly related to first-grade achievement? Sample 22 striving schools in REA (Mean FRL = 73%) 1146 first-grade children in those schools - - all children with full data *Walpole, Kaplan, & Blamey (in preparation).

Data Standardized measures of decoding and comprehension, DIBELS LNF and ORF Interviews, surveys Findings All of the schools were relatively successful; ORF mean, grade 1 = 60 wcpm Standardized scores above 50th percentile! Assessment-based planning explained a small but significant amount of the variance in first- grade achievement. This is a school factor!

So What IS Assessment-Based Planning Anyway? Schools with higher achievement were rated higher on these two characteristics: The leaders made thoughtful choices to purchase commercial curriculum materials to meet school-level needs. The leaders of this project designed a comprehensive assessment system that teachers used to differentiate instruction.

How Did We Find Out? We made a special rating sheet to rate the levels of implementation on all aspects of the project. We correlated those ratings with student achievement to find the ones that were most powerful.

How Did We Find Out? Four variables “survived” the correlations 1. Differentiation strategies for word recognition and fluency 2. Strategy and vocabulary instruction during read-alouds 3. Careful choice of instructional materials 4. School-level design of an assessment system linked to instruction

In our RF work... These variables correspond to: 1. Differentiated word recognition strategies for needs-based work 2. Interactive read-alouds of children’s literature (Beck + Duffy) 3. Use (or purchase) of curriculum materials based on their match with emergent achievement data 4. Selection of assessments that are really used to plan needs-based instruction

How did we find out? These 4 variables were highly correlated with one another; we combined the two leadership variables and the two differentiation variables. We controlled for LNF at January of kindergarten, we controlled for SES, and assessment-based planning was still correlated with first-grade scores at the school level.

What Do We Need to Learn? We need to see what variables are powerful in GARF We need to add the level of the teacher –Identify the characteristics we are targeting –Collect meaningful observational data on them to see whether they make a difference We need to use the data we have and get the data we need

That’s a tall order! (But we think we can tackle it together.)

From a design standpoint We are all trying to collect student data to measure the success of our programs. It does not make sense to measure program effects without measuring treatment fidelity. It does not make sense to measure treatment fidelity without observing the treatment. It does not make sense to document treatment fidelity without trying to improve it. Why?

Let’s look at the concept of “innovation configuration.” This is a way of finding out how fully we are implementing Reading First.

Innovation Configuration* Full implementation Partial implementation No Implementation The target practice is described here. A practice in between (or, more likely, several different ones) is described here. A description of a practice inconsistent with the target is described here. *Hall & Hord, 2001

Context ContextProcessContent Learning communities Leadership Resources Data-driven Evaluation Research-based Design Learning Collaboration Equity Quality teaching Family involvement

Moving NSDC's Staff Development Standards into Practice: Innovation Configurations By Shirley Hord, Stephanie Hirsh & Patricia Roy

Innovation Configuration for Teacher’s Professional Learning Level 4 Engages in collaborative interactions in learning teams and participates in a variety of activities that are aligned with expected improvement outcomes (e.g., collaborative lesson design, professional networks, analyzing student work, problem solving sessions, curriculum development).

Innovation Configuration for Teacher’s Professional Learning Level 2 Attends workshops to gain information about new programs and receives classroom-based coaching to assist with implementation of new strategies and activities that may be aligned with expected improvement outcomes.

Innovation Configuration for Teacher’s Professional Learning Level 1 Experiences a single model or inappropriate models of professional development that are not aligned with expected outcomes.

Procedure for Making an IC 1. Designer of an innovation describes ideal implementation of various components 2. Those “ideals” are compared with “real” implementation through observation 3. The “reals” are lined up from least like the ideal to most like the ideal 4. Then the IC can be used for observations, and even linked to student achievement!

Here are our classroom-level “ideals” for GARF so far.

Physical Environment The classroom is neat, clean, and organized so that the teacher can monitor all children and accomplish whole-group and needs-based instruction and so that children can get the materials they need. Wall space is used to display student work and curriculum-related materials that children need to accomplish tasks.

Curriculum Materials There is one core reading program in active use. There is physical evidence of coherence in the text-level and word-level skills and strategies targeted in the classroom environment. Texts and manipulatives for whole-group, small-group, and independent practice are organized and available.

Children’s Literature There is a large classroom collection of high-quality children’s literature deliberately organized and in active use that includes narratives, information texts, and multicultural texts.

Instructional Schedule There is a posted schedule inside and outside the classroom to define an organized plan for using curriculum materials for whole-group and needs-based instruction; teacher and student activities correspond to the schedule

Assessment System There is an efficient system for screening, diagnosing specific instructional needs, and progress-monitoring that is visible to the teacher and informs instructional groupings, the content of small-group instruction, and a flexible intervention system. All data are used to make instruction more effective.

Whole-Group Instruction Whole-group instruction is used to introduce new concepts and to model strategies. Children have multiple opportunities to participate and respond during instruction.

Small-Group Instruction Small-group instruction is used to reinforce, reteach, and review. Each child spends some time in small group each day; and small-group instruction is clearly differentiated. Children have multiple opportunities to participate and respond during instruction.

Independent Practice Children work alone, in small groups, or in pairs to practice skills and strategies that have been previously introduced. They read and write during independent practice. They do this with a high level of success because the teacher organizes independent practice so that it is linked to whole-group and small- group instruction.

Management The classroom is busy and active, but focused on reading. Classroom talk is positive and academic, including challenging vocabulary. Children know how to interact during whole-class, small- group, and independent work time. Very little time is spent teaching new procedures.

There’s a Lot We Don’t Know! Is it possible to observe with one form across all grades? How could we collect these observational data reliably and efficiently? Which of these might explain variance in student achievement?

Procedure for Making an IC 1. Designer of an innovation describes ideal implementation of various components 2. Those “ideals” are compared with “real” implementation through observation 3. The “reals” are lined up from least like the ideal to most like the ideal 4. Then the IC can be used for observations, and even linked to student achievement!

Professional Support System* *Joyce & Showers, 2002 Why?

You be the rater! We will give you the IC that we made for the Georgia REA study. Think about your first-grade team last year. Provide us a realistic reflection on where you stood during Year 2. We will use that data to analyze the Year 2 scores and provide you an update at our next meeting!

References Hall, G. & Hord, S. (2001). Implementing change: Patterns, principles, and potholes. Boston: Allyn and Bacon. Joyce, B., & Showers, B. (2002). Student achievement through staff development. White Plains, NY: Longman. Mosenthal, J., Lipson, M., Torncello, S., Russ, B., & Mekkelsen, J. (2004). Contexts and practices of six schools successful in obtaining reading achievement. The Elementary School Journal, 104, Taylor, B. M., Pearson, P. D., Clark, K. M., & Walpole, S. (2000). Effective schools and accomplished teachers: Lessons about primary-grade reading instruction in low-income schools. The Elementary School Journal, 101, Tivnan, T., & Hemphill, L. (2005). Comparing four literacy reform models in high-poverty schools: patterns of first-grade achievement. The Elementary School Journal, 105, Walpole, S., Kaplan, D., & Blamey, K. L. (in preparation). The effects of assessment-driven instruction on first-grade reading performance: Evidence from REA in Georgia.