1 Research on Classroom Assessment Susan M. Brookhart, Ph.D. Duquesne University and Brookhart Enterprises, LLC Montana Office of Public Instruction Conference: “There’s More to Assessment than Testing” Helena, Montana May, 2006

2 Previous reviews of literature  Bloom, 1984  Natriello, 1987  Crooks, 1988  Tittle, 1994  Black & Wiliam, 1998  Shepard, 2001  Brookhart, 2004

3 Method for this review  Built on Brookhart, 2004  ERIC search 2002-present  Hand-searched assessment journals  Colleague recommendations  Relevant references from other articles  Could have more sources, depending on what discipline(s) are searched with what definition(s) of formative assessment; suggestions welcome!  Final bibliography for this paper >100 sources

4 Organization of findings  Information about the learning process  That teachers can use for instructional decisions  And students can use for improving their own performance  That motivates students  To achieve

5 Information What information is gathered about the learning process in classrooms?  Formative-summative mix: Grades are an important reason for assessment and limit the kind of assessment teachers do (Barnes, 1985; Kusch, 1999; Schmidt & Brosnan, 1996; Wilson, 1990).

6 Information  Elementary teachers use more varied assessments methods than secondary, including “academic enablers” like effort and improvement, and rely extensively on observation (Adams & Hsu, 1998; Cizek, Fitzgerald, & Rachor, 1995; Gipps, McCallum, & Hargreaves, 2000; Gullickson, 1985; McMillan, Myron, & Workman, 2002; Nicholson & Anderson, 1993; Wilson, 1990).

7 Information  Secondary teachers use fewer commercially prepared tests, more teacher-made tests (esp. objective-Social Studies use more constructed response); recall questions dominate but thinking questions also used; consideration of “academic enablers” varies by ability level of class (Cizek, Fitzgerald, & Rachor, 1995; Frisbie, Miranda, & Baker, 1993; Gullickson, 1985; Kahn, 2000; McMillan, 2001; Stiggins & Bridgeford, 1985; Stiggins & Conklin, 1992; Wilson, 1990; Zhang & Burry-Stock, 2003).

8 Information What is the quality of this information (reliability, validity, usefulness)?  Teachers are not knowledgeable about what measurement theorists would consider relevant reliability and validity principles (Campbell & Evans, 2001; Impara, Plake, & Fager, 1993; Mertler, 2001; many studies from the 1980’s)

9 Information  Some interesting validity and reliability issues surface. Interchangeability of measures (Cizek, Fitzgerald, & Rachor, 1995) ? What is construct-relevant (Parkes, 2000) ? Students answer test questions based on relevant classroom info & experiences (30-50%) and by deduction from related experience (15- 24%) (Nuthall & Alton-Lee, 1995). Observation is part of teaching; teachers don’t separate themselves from their observations (Bulterman-Bos, Terwel, Verloops, & Wardekker, 2002). Task-specific rubrics most reliable at first (Marzano, 2002) but should that be the deciding factor (Arter & McTighe, 2001) ?

10 Teachers - Assessment beliefs  Affect assessment practices and uses (Frary, Cross, & Weber, 1993; Kahn, 2000; Mavrommatis, 1997; Raveaud, 2004; Schmidt & Brosnan, 1996; Thomas & Oldfather, 1997).  Are negative about some aspects (Barnes, 1985; Frary, Cross, & Weber, 1993; Pryor & Akwesi, 1998).  Can be altered with study (Goldberg & Roswell, 1999; Johnson, Wallace, & Thompson, 1999; Torrance & Pryor, 2001).

11 Teachers - Feedback  Types of feedback (Tunstall & Gipps, 1996). Note that the more useful feedback for improvement of work (Sadler, 1983, 1989) would be descriptive feedback.

12 Typology of Teacher Feedback (Tunstall & Gipps, 1996, p. 394) Type AType BType CType D RewardingApprovingSpecifying attainment Constructing achievement RewardsPositive personal expression Specific acknowledgement of attainment Mutual articulation of achievement 1 Positive Feedback Warm expression of feeling Use of criteria in relation to work behavior; teacher models Additional use of emerging criteria; child role in presentation 1 Achievement feedback General praiseMore specific praise Praise integral to description Positive non- verbal feedback PunishingDisapprovingSpecifying improvement Constructing the way forward 2PunishmentsNegative personal expression Correction of errors Mutual critical appraisal 2 Negative Feedback Reprimands; negative generalisations More practice given; training in self-checking Provision of strategies Improvement feedback Negative non- verbal feedback EvaluativeDescriptive

13 Teachers - Feedback  To be effective for learning (formative), feedback should be informational (drawing students’ attention to correctable errors and other salient, actionable aspects of the work), sufficiently frequent, and motivational (encouraging learners’ thoughtful reception and use of the information)  It is difficult for teachers to do this well. (Bangert-Drowns, Kulik, Kulik, & Morgan, 1991; Butler, 1987; Butler & Nisan, 1986; Elawar & Corno, 1985; Isaacson, 1999; Kluger & DeNisi, 1996; Rea-Dickens, 2001; Ryan, Connell, & Deci, 1985; Torrance & Pryor, 1998)

14 Teachers - Decisions  Teachers can learn to use assessment information for good instructional decisions, but it does not happen automatically (Bachor & Baer, 2001; Dassa, 1990; Kusch, 1999; Moreland & Jones, 2000).

15 Teachers – Environment  Classroom assessment environment (Stiggins & Conklin, 1992) is largely under the teacher’s control and affects learning and motivation (Brookhart et al., 1997, 1999, 2003).  CA environment affects graded performance by affecting achievement goals. CA environment affects intrinsic motivation both directly and by affecting achievement goals (Church, Elliott, & Gable, 2001).

16 Teachers – Environment  Sociologists have studied classroom structure (level of task differentiation, amount of student autonomy, grouping practices, grading practices). Classroom structure affects students’ ability perceptions (Rosenholtz & Rosenholtz, 1981; Simpson, 1981).  Student disengagement from high school is related to an environment where evaluations are contradictory, uncontrollable (by student), unpredictable, or unattainable (Natriello, 1996).

17 Students  Successful students do regular, ongoing self-assessment (Brookhart, 2001).  Lots of variation in student perceptions of, and responses to, assessments (Klenowski, 1995; Moni, van Kraayenoord, & Baker, 2002).  Developmental aspects of student participation in assessment (Higgins, Harris, & Kuehn, 1994; Ross, Rollheiser, & Hogaboam-Gray, 2002).

18 Motivation  Motivation has been studied as both predictor and as the thing predicted.  Learning goal orientations related to processing of info, effort, and exam performance (Elliott, McGregor, & Gable, 1999).  Affect mediates effects of learning goals on performance (Turner, Thorpe, & Meyer, 1998). Implication for formative assessment: failure-avoiding student is in the “negative affect after failure” space when s/he receives feedback.

19 Motivation  Self-regulation, as well as cognitive strategy use, is required for achievement (Pintrich & DeGroot, 1990).  Differential motivational value of different types of assessments (Brookhart & Durkin, 2003; Meece & Miller, 1999; Stefanou & Parkes, 2003). Mixed results – probably because of differences in context and nature of feedback.

20 Achievement  Studies of effects of formative assessment practices on formal “achievement” measures.  Effect sizes similar to those reported by Black & Wiliam (1998) in their review:.40 to.70.

21 Achievement  Studies using large databases LSAY – Classroom assessment variables added to explanation of science & math achievement (Brookhart, 1997) TIMMS - Classroom assessment variables interacted with self-efficacy to affect achievement; other complex relationships (Rodriquez, 2004).

22 Achievement  5 th and 6 th graders trained on how to do self-evaluation had higher classroom math achievement, ES =.40 (Ross, Hogaboam-Gray, & Rollheiser, 2002).

23 Achievement  Studies of whether challenging intellectual work and formative assessment in the classroom would be related to achievement as measured by external tests Primary (Meisels, Atkins-Burnett, Xue, & Bickel, 2003) Elementary (Newman, Bryk, & Nagaoka, 2001) High school (Wiliam, Lee, Harrison, & Black, 2004)

24 Achievement  Meisels et al. (2003): Reading ES =.68 same school, 1.60 matched schools; Math ES =.20 same school,.76 matched schools. Authors speculated the reason might have had to do with formative assessment being related to better instruction and also getting students familiar with standards and how to progress toward them.

25 Achievement  Newman, Bryk, & Nagaoka (2001): Authentic intellectual work associated with better performance on ITBS and IGAP (Reading ES =.43, Math ES =.64, Writing ES =.52). Authors speculated authentic intellectual work fostered vocabulary (and therefore concept) development and enhanced motivation.

26 Achievement  Wiliam et al. (2004). Local teachers developed formative assessment plans. Achievement was measured on external or course exams; comparisons were to different classes or different students. ES were mostly.20 to.30, median ES =.27, mean ES =.34.

27 Conclusions - Solid  Conventional assessment practices have been well documented.  Teacher beliefs about assessment affect practices.  Classroom assessment environment exists, has impact.  Certain characteristics important for effective classroom formative assessment.  Student involvement in own assessment produces learning.

28 Characteristics important for effective formative assessment  Effective FA blends assessment and instructional functions.  Clear criteria are crucial to the process.  Teacher feedback is critical.  Effective feedback compares student performance to known criteria, in relatively frequent, manageable increments.

29 Conclusions - Provisional  Teachers consider “academic enablers” in a different class from other “non- achievement” factors, and thus fair game for FA.  Relationship of classroom management to classroom assessment  Causal progression of students’ comprehension of feedback, and disposition and ability to use it  Developmental progression of same  Differential motivational value of FA methods

30 Conclusions - TBD  Application of validity and reliability theory to the way teachers do real formative assessment (and vice versa)  Specific good ways to improve teacher FA skills  Specific good ways to improve student self-assessment skills

31