1. Rubrics
Module 4
Activity 4

2. Overview
This presentation is about rubrics. It discusses the following aspects: Role of rubrics Components of rubrics Types of rubrics Selection of rubrics

3. Rubrics
First, what is a rubric? “Speaking scores express how well the examinee can speak the language being tested. They usually take the form of numbers, but they may also be verbal categories such as ‘excellent’ or ‘fair.’ In addition to the plain score, there is usually a shorter or longer statement that describes what each score means, and the series of statements from lowest to highest constitutes a rating scale [rubric].” (Luoma, 2004, p. 59)

4. Rubrics: Example Here is a very simple example of a ratig rubric: 3
Excellent
Student is able to read a paragraph with accurate intonation and pronunciation. 2
Good
Student is able to read a paragraph with generally accurate intonation and pronunciation. 1
Poor
Student, while reading a paragraph, uses generally inaccurate intonation and pronunciation.

5. Rubrics
During this presentation, please keep these questions in mind: How can you use these concepts in your teaching? How can you use these concepts in your assessments? How can you share this information with your colleagues?

6. Role of rubrics

7. Brainstorm
Before we start, please take a few minutes to think about how we rate speaking. How do you score the speaking ability of your students? Write down the process that you go through to score your students.

8. Brainstorm: Feedback
Here are the steps that are considered best practice:
Choose the type of rubric
Find a rubric or design a rubric
Share the rubric with students
Choose how many raters (one, two, three)
Train the rater(s)
Rate a live speaking performance or rate a delayed speaking performance (a recording)

9. Brainstorm
Why would you use rubrics with your students? Take a minute and write down all the reasons that you can think of.

10. Brainstorm: Feedback Rubrics can be used for a variety of reasons.
1. It makes rating less subjective and more accurate.
It shows students the characteristics that they should have in their speaking in order to excel.
It can be used as a diagnostic instrument, so that teachers can see the areas where students need help.
It can help rank students into categories, for example beginning, intermediate, and advanced.

11. Components of rubrics

12. Preview
Let’s look at another speaking rubric. How many components do you see?
Adapted from The Test of English for Educational Purposes fluency rubric (Weir, 1993:44)
Utterances halting, fragmentary and in coherent 1
Utterances hesitant is often incomplete. Sentences are, for the most part, disjointed and restricted in length. 2
Signs of developing attempts at using cohesive devices, especially conjunctions. Utterances may still be hesitant, but are gaining in coherence, speed, and length. 3
Utterances, whilst occasionally hesitant, are characterized by an evenness and flow. Fillers are effectively used.

13. Preview
Adapted from The Test of English for Educational Purposes fluency rubric (Weir, 1993:44)
Utterances halting, fragmentary and in coherent 1
Utterances hesitant is often incomplete. Sentences are, for the most part, disjointed and restricted in length. 2
Signs of developing attempts at using cohesive devices, especially conjunctions. Utterances may still be hesitant, but are gaining in coherence, speed, and length. 3
Utterances, whilst occasionally hesitant, are characterized by an evenness and flow. Fillers are effectively used.
I see three.

14. Components of rubrics
Here the three important elements when creating a rating rubric: Sub-Construct Score Scale definition / scale descriptor

15. Components of rubrics
Sub-construct refers to the skills or abilities that the speaking task tries to measure. For example: Fluency Intelligible pronunciation Knowledge of relevant vocabulary Accurate knowledge of grammar

16. Components of rubrics
The score is awarded based on the performance of the student. Each level has a corresponding score.

17. Components of rubrics
Scale Definition / Scale descriptor refers to the specific features at each level of the rubric The descriptors should indicate a progression as you move up the rubric.
Adapted from The Test of English for Educational Purposes fluency rubric (Weir, 1993:44)
Utterances halting, fragmentary and in coherent 1
Utterances hesitant is often incomplete. Sentences are, for the most part, disjointed and restricted in length. 2
Signs of developing attempts at using cohesive devices, especially conjunctions. Utterances may still be hesitant, but are gaining in coherence, speed, and length. 3
Utterances, whilst occasionally hesitant, are characterized by an evenness and flow. Fillers are effectively used.

18. Components of rubrics
Depending on the types of rubric, a single score or multiple scores can be given according to test takers’ performance. We will return to this topic later.

19. Review
Adapted from The Test of English for Educational Purposes fluency rubric (Weir, 1993:44)
Utterances halting, fragmentary and in coherent 1
Utterances hesitant is often incomplete. Sentences are, for the most part, disjointed and restricted in length. 2
Signs of developing attempts at using cohesive devices, especially conjunctions. Utterances may still be hesitant, but are gaining in coherence, speed, and length. 3
Utterances, whilst occasionally hesitant, are characterized by an evenness and flow. Fillers are effectively used.
Do you remember? Where are the scale definition/scale descriptor, sub-construct, and score?

20. Activity: Feedback
scale definition/ scale descriptor: specific features to be rated in the rubric are coherence, cohesive devices, speed, length, pace etc. Sub-construct: the ability to speak fluently. Score: the rubric uses rubric scores from 0 to 4.

21. Types of rubrics

22. Reflect
Think about the rubrics you have used before. What did they look like?

23. Rating rubrics: Types
There are different rubrics. Here are two common ones: Holistic rubric Analytic rubric

24. Types of rubrics Holistic rating rubric:
All criteria being evaluated are considered together (e.g.,
delivery, language use, topic development are all considered
together)
2. A single score is awarded.

25. Types of rubrics
Holistic rubric: sample holistic rating rubric Please read the document “ACTFL Speaking Scale Descriptors” To access this document, leave the PPT and go to the next content page titled “VI.a: For Assignment 2, READ “Speaking Scale Descriptors”. After reading the rubric, return to this PPT.

26. Activity
Look at the “ACTFL Speaking Scale Descriptors.” How do the descriptors change for each level? What can examinees at each level do in terms of topic development? In other words, what topics can examinees at each level discuss? For example, at the Distinguished level, examinees can “can reflect on a wide range of global issues and highly abstract concepts in a culturally appropriate manner.” Take a few minutes and locate the references to topics in the rubric.

27. Activity: Feedback
Here’s what we found for topic: Distinguished: able to reflect on a wide range of global issues and highly abstract concepts in a culturally appropriate manner. Superior: able to communicate with accuracy and fluency in order to participate fully and effectively in conversations on a variety of topics in formal and informal settings from both concrete and abstract perspectives. Advanced High: able to handle the tasks pertaining to the Superior level but cannot sustain performance at that level across a variety of topics.

28. Activity: Feedback
Advanced Mid: able to participate actively in most informal and some formal exchanges on a variety of concrete topics relating to work, school, home, and leisure activities, as well as topics relating to events of current, public, and personal interest or individual relevance. Advanced Low: able to participate in most informal and some formal conversations on topics related to school, home, and leisure activities. They can also speak about some topics related to employment, current events, and matters of public and community interest.

29. Activity: Feedback
This demonstrates how the descriptors change as you move across the levels of the rubric. Also, note that the same feature (topic) was addressed within every band in the rubric. This is another important characteristic of rubrics.

30. Types of rubrics Analytic rubric:
Each rubric represents multiple underlying sub-constructs.
Multiple scores are awarded.

31. Rating rubrics: Types
Analytic Rating rubric: Sample analytic rating rubric
Please read the document “IELTS Speaking Scale Descriptors.”
To access this document, leave the PPT and go to the next content page titled “VI.b: For Assignment 2, READ “IELTS Speaking band descriptors”.
After reading the rubric, return to this PPT.

32. Rating rubrics: Types Analytic rubric: Score Calculation
A student will get a score out of 9 for each sub-construct. The scores are added together and then that score is divided by 4.
For example, if a student gets a 7 for Fluency and Coherence, a 6 for Lexical Resource , a 7 for Grammatical range and accuracy, and an 8 for pronunciation, the total is 28, divided by 4, he/she will get a 7 overall for his/her speaking.

33. Activity
Look at the sub-construct fluency in the “IELTS Speaking Scale Descriptors” speaking rubric and see how it changes for each score.

34. Activity: Feedback
Here’s what we saw: Band 9: speak fluently with only rare repetition or self correction Band 8: speaking fluently with only occasional repetition or self- correction Band 7: speak at length without noticeable effort or loss of coherence Band 6: is willing to speak at length, though may lose coherence at times due to occasional repetition, self –correction or hesitation

35. Activity: Feedback
Band 5: maintain flow of speech but uses repetition, self- correction and/or slow speech to keep going
Band 4: cannot respond without noticeable pauses and may speak slowly, with frequent repetition and self-correction
Band 3: speak with long pauses
Band 2: pause lengthily before most words
Band 1: no communication possible
Band 0: does not attend

36. Types of rubrics
So, which type of rubric is better? Analytic or Holistic? It depends…

37. Types of rubrics
Comparison of holistic and analytic rubrics (Weigle, 2002)
Quality
Holistic
Analytic
Reliability
Lower (most of the time)
Higher
Construct Validity
It assumes all relevant aspects of abilities develop at the same rate
It is more appropriate for L2 learners
Practicality
Fast and easy
Time-consuming
Impact
A single score may be misleading
It can provide more useful diagnostic information
Authenticity
Reading a rubric holistically is a more natural process
Raters may tend to read holistically and adjust analytical scores to match holistic impressions

38. Rating rubrics: Types
So, which type of rubric is better? Analytic or Holistic? It depends on the test purpose, how you will use the test results, and your available resources. As an example, if you want to provide students with diagnostic information so that they know what areas they need to improve, an analytic rubric would be more appropriate.

39. Selection of rubrics

40. Reflect
When rating students’ speaking performance, do you generally use holistic or analytic rubrics? Why?

41. Selecting rubrics
There are several aspects that should be considered when selecting a rubric.

42. Selecting rubrics
1.Identify the learning outcomes before selecting the rubric. 2.Select the type of rubric that is most appropriate for the task and the purpose of the assessment. 3.Rubric characteristics should be directly observable and clearly defined.

43. Selecting rubrics
4. Having between three and seven rating positions should yield
scores that better reflect a student’s ability.
5. You may want features of speaking that you explicitly teach in
your class included in the rating rubric (e.g. fluency, stress,
rhythm/intonation). Consider these features when you are
selecting your speaking rubric.

44. Activity
Given the following scenario, which type of rubric would you use, holistic or analytic?
The focus is on a specific speaking ability (e.g. fluency).
The speaking task is a formative achievement test.
You are going to assess significant numbers of students (e.g. 50 students).

45. Activity: Feedback
I would use a holistic rubric for the first and third tests.
1. The focus is on a specific speaking ability (e.g. fluency).
Holistic rubrics measure a specific domain of knowledge instead of multiple skills.
2. The speaking task is a formative achievement test.
Analytic rubrics include many sub-constructs.
3. You are going to assess significant numbers of students (e.g. 50 students).
Holistic rubrics are a quick way to grade 50 students’ speaking performance.

46. Practice
Now, let’s practice creating a prompt and a rubric. This is an offline, non-graded assignment. For this assignment you will write a speaking task. Be sure to describe the construct you are testing. Then, write the speaking prompt. Use one of the sample rubrics from this lesson. Pay attention to why you are choosing the holistic or analytic rubric. The next slide lists elements that you should include.

47. Practice
characteristics of test-takers (age, L1, English ability, anything else relevant)
construct
directions (tell the student exactly what to do, using the simplest language possible)
prompt
scoring rubric

48. Practice Speaking Characteristics of test-takers Construct/subskills
Construct/subskills
Directions
Prompt
Scoring Rubric

49. Key Terms & Concepts
New Terms: Score Rubric Scale Descriptor Holistic rubric Analytic rubric

50. Next Steps
Now, you should do assignment #2. In this assignment, you will update the glossary terms.

51. Sources
ACTFL. (2012). ACTFL Proficiency Guidelines Retrieved from manuals/actfl-proficiency-guidelines-2012/english/speaking IELTS. (2016). IELTS Speaking Band Descriptors (Public Version). Retrieved from IELTS_Speaking_band_descriptors.pdf Luoma, S. (2004). Assessing Speaking. Cambridge University Press. Weigle, S.C.(2002). Assessing Writing. Cambridge University