1 Innovation in Assessment and Evaluation Prof. dr. Martin Valcke Ghent University Maputo July 2011

Structure Advance organizer Traditional approaches Trends in assessment & evaluation Assumptions in relation to these trends Research Practices Conclusions 2

3

Advance organizer

5 Transmission of your message 3. Eye Contact

6 Can I learn this?

The importance of assessment & evaluation

8

Traditional approaches 9

Critical issues Validity of evaluation approach in view of assessment of skills and complex knowledge Fant et al., (1985) Rating scales, daily logs, anecdotal records, behavior coding, and self-assessment for evaluating student teachers. Oral examinations, portfolio assessment, central assessment centres, 360° assessment –… 10

Recent developments Individual learner Group learner External institution Teachers Expert eacher Assessment system Institutional level

Recent developments Stronger focus on “consequential validity”of measurement (Gielen, Dochy & Dierick, 2003) Stronger emphasis on feedback value of assessment What is the “learning potential” of the assessment approach 12

Recent developments Stiggins (1987): performance assessment Performance assessment is expected to be geared in a better way to assess complex behavior in medical, legal, engineering, … and educational contexts (Sluijsmans, et.al., 2004). 13

Concrete examples Self- and peer assessment Rubrics based assessment 14

Self- and peer-assessment 15

How to help the student to assess him/herself? 16

Self- and peer assessment Learn about your own learning process. Schmitz (1994): “assessment-as-learning”. ~ self corrective feedback 17

See experiential learning cycle of Kolb. Boekaerts (1991) self evaluation as a competency. Development of metacognitive knowledge and skills (see Brown, Bull & Pendlebury, 1998, p.181). Freeman & Lewis (1998, p.56-59): developing pro-active learners 18

19 The Learning Cycle Model

Is it possible? 21 Group evaluations tend to fluctuate around the mean

Learning to evaluate Develop checklists Give criteria Ask to look for quality indicators. Analysis of examples good and less good practices: develop a quality “nose” 22

Learning to evaluate Freeman & Lewis (1998, p.127) : Learner develops list of criteria. Pairs of learners compare listed criteria. Pairs develop a criterion checklist. Individual application of checklist. Use of checklist to evalute work of other learner. Individual reworks his/her work. Final result checkeed by teacher and result compared to learner evaluation. Pairs recheck their work on the base of teacher feedback. 23

Learning to evaluate Peer evaluation is not the same as Peer grading Final score is given by teacher Part of score could build on accuracy of self/peer evaluation and self- correction 24

25

Rubrics 26

Rubrics Rubrics focus on the relationship between competencies, criteria, and indicators and are organized along mastery levels (Morgan, 1999). 27

28

Rubrics Rubric: scoring tool for a qualitative assessment of the quality level of an authentic or complex activity –A rubric builds on criteria, enriched with a scale to indicate a mastery level. –For each level, standards are indicated that reflect this level. –A rubric dictates both teacher and student what is concretely expected. –Rubrics are used for “high stake assessment” and “formative assessment” (Arter & McTighe, 2001; Busching, 1998; Perlman, 2003). Rubrics focus on the relationship between competencies, criteria, are organized along mastery levels (Morgan, 1999). 29

Rubrics: indicator-based assessment Assessment objective –Criteria Enriched with indicators in terms of observable behavior Limited number of indicators 30

Critical thinking rubric 31

Assumptions about rubrics Larger consistency in scores (reliability). More valid assessment of complex behavior. Positive impact on related learning process. 32

Critical issues Adoption of this assessment approach is marred by teacher beliefs about nature of evaluation (see e.g., Chong, Wong, & Lang, 2004); Also student beliefs (Joram & Gabriele, 1998) Validity of the criteria and indicators (Linn, 1990), Reliability of performance evaluation, e.g., when multiple evaluators assess and score performance (Flowers & Hancock, 2003). 33

Research about rubrics use Review article 75 studies about rubrics usage Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2, 130– 144. –(1) the reliable scoring of performance assessments can be enhanced by the use of rubrics, especially if they are analytic, topic-specific, and complemented with exemplars and/or rater training; –(2) rubrics do not facilitate valid judgment of performance assessments per se. However, valid assessment could be facilitated by using a more comprehensive framework of validity; –(3) rubrics seem to have the potential of promoting learning and/or improve instruction. The main reason for this potential lies in the fact that rubrics make expectations and criteria explicit, which also facilitates feedback and self-assessment. 34

Conditions effective usage Check frame of reference for the rubric: tasks, objectives Train the users Use multiple assessors: interrater usage Developed by teacher and/or students! 35

Development of rubrics Choose quality criteria: 4 to 15 statements describing the nature of a criterion Determine bandwidth to judge differences in quality related to the criterion: e.g., 0-5 or qualitative descriptors Elaborate descriptors for each bandwidth level: concrete operational terms Start from available student work! 36

Writing a fiction story Complex skill Criteria? Rubrics: example

Rubrics

39

Rubrics Rubric: scoringstool voor een kwalitatieve inschatting van het niveau van een authentieke of complexe activiteit. –Een rubriek bouwt verder op criteria die verrijkt worden met een schaal waarop beheersingsniveaus zijn aangegeven –Per beheersingsniveau worden standaarden aangegeven die die niveau weerspiegelen. –Een rubric geeft zowel voor de lesgever als de student aan wat concreet verwacht wordt. –Rubrics worden voor “high stake assessment” gebruikt en voor “formatieve toetsing” (in functie van leren). (Arter & McTighe, 2001; Busching, 1998; Perlman, 2003). Rubrics focus on the relationship between competencies- criteria, and indicators and are organized along mastery levels (Morgan, 1999). 40

Performance assessment Rubrics focus on the relationship between competencies-criteria, and indicators and are organized along mastery levels (Morgan, 1999). 41

Aanpak ontwikkeling rubric Kies criteria voor verwacht gedrag –4 tot 15 statements die criterium beschrijven Bepaal bandbreedte die verschil in bereiken criterium weergeven –Bijv. 0-5 of kwalitatieve termen Werk een beschrijving uit voor elke waarde in de bandbreedte –Concreet observeerbare/vaststelbare kwalificaties 42

43

44

More information? Overview of tools, examples, theory, bacdkground, research: Critical thinking rubrics: l Rubric generators: Interesting rubric sites: Rubric APA research paper: K12 examples: General intro and overview: 45

46

Statements about evaluation Learners should be trained to develop themselves such rubrics. Staff should collaborate in developing formal assessment and summative assessment rubrics Rubrics will help staff to be more concrete as to their teaching and learning focus 47

48 Innovation in Assessment and Evaluation Prof. dr. Martin Valcke Ghent University Maputo July 2011