1. Designing Rubrics For Classroom Assessment Professor Timothy Farnsworth, CUNY Hunter College

2. Communicative Assessment Is Hard!  We are making inferences about what people can and can’t get done in the real world, not:  What grammar or vocabulary they can choose from 4 options on a MC test  Tasks must be realistic but can’t be TOO realistic:  Time constraints  Stakes are different  Interlocutors are different  Setting, format usually different 2

3. Speaking Assessment Challenges  Speaking is the hardest skill to assess  Impermanent -> must be rated on the spot or recorded  Interactive -> Real communication takes two persons  Nonverbal -> Much real communication takes place without language use  Topical -> Students’ knowledge base affects their speaking output 3

4. Oral Assessment / Testing Tasks  Must get students involved in speaking!  In communication, many things happening  Listening  Nonverbal channeling  Decisions made about what to say, what not to say  Students practice avoidance strategies, game the system, don’t take risks  Assessment tasks take many forms 4

5. Types of Assessment Tasks  Many options – some common ones:  Interview, either scripted or unscripted  Picture description  Role Play  Pair or group tasks  Almost any classroom task, really! 5

6. Sample task: Pair oral assessment  Each student has a different picture; communicate to find similarities and differences 6

7. Scoring tasks and communicating results  We need some way to turn the “observation” into a “score” or “grade”  Usually done via numbers, can be done as letters, smiley faces, etc. – doesn’t matter  Should be clear, simple  Should be “reliable” -> repeatable, dependable  Should be “valid” -> meaningful  Shouldn’t cause any harm 7

8. Types of scoring rubric  Holistic  Analytic  Task Fulfillment (“primary trait”)  Performance decision trees  Checklists 8

9. Holistic Rubrics  One overall score tied to a “performance descriptor”  Most familiar type of rubric  Assign score of the description that best matches performance  Problem: Performance doesn’t match description at the same level in all areas  Advantage: Simple to use and understand 9 ScoreDescription 60Communication almost always effective: task performed very competently Functions performed clearly and effectively Appropriate response to audience/situation Coherent, with effective use of cohesive devices Use of linguistic features almost always effective; communication not affected by minor errors 50….. 40Communication somewhat effective: task performed somewhat competently Functions performed somewhat clearly and effectively Somewhat appropriate response to audience/situation Somewhat coherent, with some use of cohesive devices Use of linguistic features somewhat effective; communication sometimes affected by errors 30… 20No effective communication: no evidence of ability to perform task No evidence that functions were performed …

10.  Two or more categories of scores  Usually based around linguistic categories such as pronunciation, grammar, etc.  Also quite common  Assign scores for each category per performance  Problem: Can be cognitively challenging for raters  Advantage: provides more detailed information LevelAccuracyFluencyPronunciation 0No structured response; single words only Speech is halting, long pauses Wholly or partly unintelligible due to pron. Errors 1Multiple inaccuracies in vocabulary choice, sentence structure Pauses occur but are not overly long, speaker does not inordinately ‘word search’ Pron. Errors increase difficulty for hearers 2Few major errors wrt vocab or SS Normal or near- normal speed and flow of speech Pronunciation is native-like or does not increase difficulty for listeners Analytic Rubrics 10

11. Task Fulfillment Rubrics  Usually look like holistic rubrics  Instead of describing language, they describe whether the student completed the “task”  Usually used in role play type assessments  Useful in testing for employment, other settings  Useful when there is a specific set of tasks that involve language and can be specified  Example rubric for hotel employee testing:  Job applicant is tested for ability to respond to guest requests and complaints, answer telephone, etc. in role play situation, 11 ScoreDescription 4Candidate was polite, friendly, helped guest successfully, responded to complaints appropriately, asked for clarification, etc. 3… 2… 1Candidate appeared rude, could not help guest successfully, did not respond to complaints appropriately, could not ask for clarification, etc. 0No response

12. Performance Decision Trees  Fairly new to the scene (Fulcher 2011)  Use a flowchart design  Good for clear, simple scoring when the task is very consistent / clear  Give points for each successful “yes” on the flowchart  Lets the test raters answer very specific questions 12

13. 13

14. Self / Peer Assessment 14  Students have some good ideas about their and their classmate’s performances  In a busy classroom, teacher cannot hear everyone  Students can use any feedback they get  Training students to listen carefully and help each other is good practice in general  But: Students are not always good evaluators of their peers  May not catch pronunciation or other important mistakes  May be LESS proficient than the person they are supposed to assess  Only ask them to assess in areas they can definitely “do” – yes / no checklists are great for this

15. Checklist Rubrics  Phrased very simply like a can-do statement:  “Student can understand and use vocabulary for classroom objects. Yes / No / Partially  “Student can use formal language in giving oral presentations” Yes / No / Partially  Can be broken into parts:  “Student understands WH questions” Yes / No / Partially  “Student responds to WH questions using appropriate grammar and vocabulary” Yes / No / Partially  “Student responds to WH questions at length and returns questions appropriately” Yes / No / Partially 15

16. Checklists for student self-evaluation  Self Evaluation – teacher collects evidence  Checklists are simple and clear (example)  List main objectives for the week, create checklist for students  Have them do and turn in anonymously  Use the info to plan review sesssions in future 16

17. Keys to consistent, fair scoring 1: KISS principle for tasks  Keep It Simple, Sweetheart!  Don’t use any more complicated assessment than you really need  One or two tasks is enough as long as you are getting the right kind of interaction  Make sure you get “enough speech” to make a rating using the right rubric  Make a system that everyone can easily understand  Good speaking testing usually can be done in less than five or ten minutes  More “high stakes” testing takes longer, up to an hour 17

18. Keys to consistent, fair scoring 2: Use the simplest rubric that works 18  I like checklists or performance trees for classroom and self assessments  I like analytic rubrics for more high-stakes testing  They give information on what things the student needs to work on  Holistic rubrics don’t provide much information for students  I don’t like task fulfillment rubrics for most situations  It’s not clear from the score what the student needs to work on  It’s sometimes possible to do a task using little language; the task fulfillment setup still allows high scores

19. Keys to consistent, fair scoring 3: Write the rubric so students can understand it 19  Use plain and simple language  Use “I can / can’t do X” or “You did X” when possible  Explain the scores and the rubric to them before the test –  no reason for it to be a secret  They should understand why they got their score and what they can work on next time  Do not use a self assessment for a grade  Do not use peer assessment for a grade

20. Your assignment: Write a scoring rubric 20  Find a partner next to you  Together, pick one rubric style: Holistic, analytic, decision tree, checklist, task fulfillment  Think about students in a typical class you teach:  What level are they?  What do you need to learn more about in terms of their speaking skills?  Write a rubric with a few levels and clear descriptions  Compare your rubric with a pair near you – does it look similar? Why or why not?

21. Sample task: Pair oral assessment  Each student has a different picture; communicate to find similarities and differences 21