Rubrics Module 4 Activity 4

Overview This presentation is about rubrics. It discusses the following aspects: Role of rubrics Components of rubrics Types of rubrics Selection of rubrics

Rubrics First, what is a rubric? “Speaking scores express how well the examinee can speak the language being tested. They usually take the form of numbers, but they may also be verbal categories such as ‘excellent’ or ‘fair.’ In addition to the plain score, there is usually a shorter or longer statement that describes what each score means, and the series of statements from lowest to highest constitutes a rating scale [rubric].” (Luoma, 2004, p. 59)

Rubrics: Example Here is a very simple example of a ratig rubric: 3 Excellent Student is able to read a paragraph with accurate intonation and pronunciation. 2 Good Student is able to read a paragraph with generally accurate intonation and pronunciation. 1 Poor Student, while reading a paragraph, uses generally inaccurate intonation and pronunciation.

Rubrics During this presentation, please keep these questions in mind: How can you use these concepts in your teaching? How can you use these concepts in your assessments? How can you share this information with your colleagues?

Role of rubrics

Brainstorm Before we start, please take a few minutes to think about how we rate speaking. How do you score the speaking ability of your students? Write down the process that you go through to score your students.

Brainstorm: Feedback Here are the steps that are considered best practice: Choose the type of rubric Find a rubric or design a rubric Share the rubric with students Choose how many raters (one, two, three) Train the rater(s) Rate a live speaking performance or rate a delayed speaking performance (a recording)

Brainstorm Why would you use rubrics with your students? Take a minute and write down all the reasons that you can think of.

Brainstorm: Feedback Rubrics can be used for a variety of reasons. 1. It makes rating less subjective and more accurate. It shows students the characteristics that they should have in their speaking in order to excel. It can be used as a diagnostic instrument, so that teachers can see the areas where students need help. It can help rank students into categories, for example beginning, intermediate, and advanced.

Components of rubrics

Preview Let’s look at another speaking rubric. How many components do you see? Adapted from The Test of English for Educational Purposes fluency rubric (Weir, 1993:44) Utterances halting, fragmentary and in coherent 1 Utterances hesitant is often incomplete. Sentences are, for the most part, disjointed and restricted in length. 2 Signs of developing attempts at using cohesive devices, especially conjunctions. Utterances may still be hesitant, but are gaining in coherence, speed, and length. 3 Utterances, whilst occasionally hesitant, are characterized by an evenness and flow. Fillers are effectively used.

Preview Adapted from The Test of English for Educational Purposes fluency rubric (Weir, 1993:44) Utterances halting, fragmentary and in coherent 1 Utterances hesitant is often incomplete. Sentences are, for the most part, disjointed and restricted in length. 2 Signs of developing attempts at using cohesive devices, especially conjunctions. Utterances may still be hesitant, but are gaining in coherence, speed, and length. 3 Utterances, whilst occasionally hesitant, are characterized by an evenness and flow. Fillers are effectively used. I see three.

Components of rubrics Here the three important elements when creating a rating rubric: Sub-Construct Score Scale definition / scale descriptor

Components of rubrics Sub-construct refers to the skills or abilities that the speaking task tries to measure. For example: Fluency Intelligible pronunciation Knowledge of relevant vocabulary Accurate knowledge of grammar

Components of rubrics The score is awarded based on the performance of the student. Each level has a corresponding score.

Components of rubrics Scale Definition / Scale descriptor refers to the specific features at each level of the rubric The descriptors should indicate a progression as you move up the rubric. Adapted from The Test of English for Educational Purposes fluency rubric (Weir, 1993:44) Utterances halting, fragmentary and in coherent 1 Utterances hesitant is often incomplete. Sentences are, for the most part, disjointed and restricted in length. 2 Signs of developing attempts at using cohesive devices, especially conjunctions. Utterances may still be hesitant, but are gaining in coherence, speed, and length. 3 Utterances, whilst occasionally hesitant, are characterized by an evenness and flow. Fillers are effectively used.

Components of rubrics Depending on the types of rubric, a single score or multiple scores can be given according to test takers’ performance. We will return to this topic later.

Review Adapted from The Test of English for Educational Purposes fluency rubric (Weir, 1993:44) Utterances halting, fragmentary and in coherent 1 Utterances hesitant is often incomplete. Sentences are, for the most part, disjointed and restricted in length. 2 Signs of developing attempts at using cohesive devices, especially conjunctions. Utterances may still be hesitant, but are gaining in coherence, speed, and length. 3 Utterances, whilst occasionally hesitant, are characterized by an evenness and flow. Fillers are effectively used. Do you remember? Where are the scale definition/scale descriptor, sub-construct, and score?

Activity: Feedback scale definition/ scale descriptor: specific features to be rated in the rubric are coherence, cohesive devices, speed, length, pace etc. Sub-construct: the ability to speak fluently. Score: the rubric uses rubric scores from 0 to 4.

Types of rubrics

Reflect Think about the rubrics you have used before. What did they look like?

Rating rubrics: Types There are different rubrics. Here are two common ones: Holistic rubric Analytic rubric

Types of rubrics Holistic rating rubric: All criteria being evaluated are considered together (e.g., delivery, language use, topic development are all considered together) 2. A single score is awarded.

Types of rubrics Holistic rubric: sample holistic rating rubric Please read the document “ACTFL Speaking Scale Descriptors” To access this document, leave the PPT and go to the next content page titled “VI.a: For Assignment 2, READ “Speaking Scale Descriptors”. After reading the rubric, return to this PPT.

Activity Look at the “ACTFL Speaking Scale Descriptors.” How do the descriptors change for each level? What can examinees at each level do in terms of topic development? In other words, what topics can examinees at each level discuss? For example, at the Distinguished level, examinees can “can reflect on a wide range of global issues and highly abstract concepts in a culturally appropriate manner.” Take a few minutes and locate the references to topics in the rubric.

Activity: Feedback Here’s what we found for topic: Distinguished: able to reflect on a wide range of global issues and highly abstract concepts in a culturally appropriate manner. Superior: able to communicate with accuracy and fluency in order to participate fully and effectively in conversations on a variety of topics in formal and informal settings from both concrete and abstract perspectives. Advanced High: able to handle the tasks pertaining to the Superior level but cannot sustain performance at that level across a variety of topics.

Activity: Feedback Advanced Mid: able to participate actively in most informal and some formal exchanges on a variety of concrete topics relating to work, school, home, and leisure activities, as well as topics relating to events of current, public, and personal interest or individual relevance. Advanced Low: able to participate in most informal and some formal conversations on topics related to school, home, and leisure activities. They can also speak about some topics related to employment, current events, and matters of public and community interest.

Activity: Feedback This demonstrates how the descriptors change as you move across the levels of the rubric. Also, note that the same feature (topic) was addressed within every band in the rubric. This is another important characteristic of rubrics.

Types of rubrics Analytic rubric: Each rubric represents multiple underlying sub-constructs. Multiple scores are awarded.

Rating rubrics: Types Analytic Rating rubric: Sample analytic rating rubric Please read the document “IELTS Speaking Scale Descriptors.” To access this document, leave the PPT and go to the next content page titled “VI.b: For Assignment 2, READ “IELTS Speaking band descriptors”. After reading the rubric, return to this PPT.

Rating rubrics: Types Analytic rubric: Score Calculation A student will get a score out of 9 for each sub-construct. The scores are added together and then that score is divided by 4. For example, if a student gets a 7 for Fluency and Coherence, a 6 for Lexical Resource , a 7 for Grammatical range and accuracy, and an 8 for pronunciation, the total is 28, divided by 4, he/she will get a 7 overall for his/her speaking.

Activity Look at the sub-construct fluency in the “IELTS Speaking Scale Descriptors” speaking rubric and see how it changes for each score.

Activity: Feedback Here’s what we saw: Band 9: speak fluently with only rare repetition or self correction Band 8: speaking fluently with only occasional repetition or self- correction Band 7: speak at length without noticeable effort or loss of coherence Band 6: is willing to speak at length, though may lose coherence at times due to occasional repetition, self –correction or hesitation

Activity: Feedback Band 5: maintain flow of speech but uses repetition, self- correction and/or slow speech to keep going Band 4: cannot respond without noticeable pauses and may speak slowly, with frequent repetition and self-correction Band 3: speak with long pauses Band 2: pause lengthily before most words Band 1: no communication possible Band 0: does not attend

Types of rubrics So, which type of rubric is better? Analytic or Holistic? It depends…

Types of rubrics Comparison of holistic and analytic rubrics (Weigle, 2002) Quality Holistic Analytic Reliability Lower (most of the time) Higher Construct Validity It assumes all relevant aspects of abilities develop at the same rate It is more appropriate for L2 learners Practicality Fast and easy Time-consuming Impact A single score may be misleading It can provide more useful diagnostic information Authenticity Reading a rubric holistically is a more natural process Raters may tend to read holistically and adjust analytical scores to match holistic impressions

Rating rubrics: Types So, which type of rubric is better? Analytic or Holistic? It depends on the test purpose, how you will use the test results, and your available resources. As an example, if you want to provide students with diagnostic information so that they know what areas they need to improve, an analytic rubric would be more appropriate.

Selection of rubrics

Reflect When rating students’ speaking performance, do you generally use holistic or analytic rubrics? Why?

Selecting rubrics There are several aspects that should be considered when selecting a rubric.

Selecting rubrics 1.Identify the learning outcomes before selecting the rubric. 2.Select the type of rubric that is most appropriate for the task and the purpose of the assessment. 3.Rubric characteristics should be directly observable and clearly defined.

Selecting rubrics 4. Having between three and seven rating positions should yield scores that better reflect a student’s ability. 5. You may want features of speaking that you explicitly teach in your class included in the rating rubric (e.g. fluency, stress, rhythm/intonation). Consider these features when you are selecting your speaking rubric.

Activity Given the following scenario, which type of rubric would you use, holistic or analytic? The focus is on a specific speaking ability (e.g. fluency). The speaking task is a formative achievement test. You are going to assess significant numbers of students (e.g. 50 students).

Activity: Feedback I would use a holistic rubric for the first and third tests. 1. The focus is on a specific speaking ability (e.g. fluency). Holistic rubrics measure a specific domain of knowledge instead of multiple skills. 2. The speaking task is a formative achievement test. Analytic rubrics include many sub-constructs. 3. You are going to assess significant numbers of students (e.g. 50 students). Holistic rubrics are a quick way to grade 50 students’ speaking performance.

Practice Now, let’s practice creating a prompt and a rubric. This is an offline, non-graded assignment. For this assignment you will write a speaking task. Be sure to describe the construct you are testing. Then, write the speaking prompt. Use one of the sample rubrics from this lesson. Pay attention to why you are choosing the holistic or analytic rubric. The next slide lists elements that you should include.

Practice characteristics of test-takers (age, L1, English ability, anything else relevant) construct directions (tell the student exactly what to do, using the simplest language possible) prompt scoring rubric

Practice Speaking Characteristics of test-takers Construct/subskills   Construct/subskills Directions Prompt Scoring Rubric

Key Terms & Concepts New Terms: Score Rubric Scale Descriptor Holistic rubric Analytic rubric

Next Steps Now, you should do assignment #2. In this assignment, you will update the glossary terms.

Sources ACTFL. (2012). ACTFL Proficiency Guidelines 2012. Retrieved from https://www.actfl.org/publications/guidelines-and- manuals/actfl-proficiency-guidelines-2012/english/speaking IELTS. (2016). IELTS Speaking Band Descriptors (Public Version). Retrieved from http://takeielts.britishcouncil.org/sites/default/files/ IELTS_Speaking_band_descriptors.pdf Luoma, S. (2004). Assessing Speaking. Cambridge University Press. Weigle, S.C.(2002). Assessing Writing. Cambridge University