1. Assessing Speaking: Developing the Test
Module 4
Activity 3

2. Review
In Assessing Speaking: First Steps, we talked about the following aspects: 1. Speaker Ability 2. Functions Involved with Speaking 3. Considerations of Speaking Assessment 4. Speaking Assessment: Where do we start?

3. Overview
In this presentation, we will talk about how to build on this information to do the following:
Develop Tasks to Assess Speaking
Write speaking prompts
Speaking Tests

4. Develop Tasks to Assess Speaking

5. Brainstorm
Take a few minutes to think about how you assess speaking. Make a list of the different ways that you assess speaking in your classroom.

6. Brainstorm: Feedback
Now compare your list to this list of a few ways to assess speaking: Description task Narrative task Instruction task Decision task Comparing and Contrasting task Explaining and Predicting task Structured Speaking task Role-play Interview

7. A closer look
Let’s look more closely at different tasks to assess speaking. As you learn about these tasks, consider: The practicality of each task Whether the task can be used in your teaching and assessing context

8. Tasks to Assess Speaking
Description task Narrative task Instruction task Decision task Comparing and Contrasting task Explaining and Predicting task Structured Speaking task Role-play Interview

9. Tasks to Assess Speaking: Description Task
In a description task, the examinee will be asked to describe something that they are familiar with. This can take place in a one-to- one interview, or in pair work. Example: The examinee is asked to describe the last vacation that they took.

10. Tasks to Assess Speaking: Narrative Task
In a narrative task, the examinee is given a series of pictures that tell a story. The examinee will look at the pictures and tell the examiner the story that the pictures show. Narrative tasks show how well the examinee can recount a sequence of events.
Example: Tell the story that the pictures show.

11. Tasks to Assess Speaking: Instruction Task
In an instruction task, the examinee delivers instructions to the examiner or a partner about how something is done. The listener may ask questions for clarification if they do not understand the directions.
Examples:
Imagine you are at home and need to get to the post office. Give instructions on how to get there.
A friend is watching your house while you are out of town. Give instructions to your friend on what needs to be done to take care of your cat.

12. Tasks to Assess Speaking: Comparing and Contrasting Task
Comparing and contrasting tasks are designed to assess the examinees ability to describe and also compare two things. This often means using comparatives and superlatives (more, less; endings: -er, -est). These tasks can be done alone, or with a partner. Tasks may also include pictures.
Examples:
Look at the pictures and compare and contrast the level of danger in the extreme sports shown.
Describe the differences between living in a city and living in rural areas.

13. Tasks to Assess Speaking: Decision Task
Decision tasks are usually performed in pairs. The examinees are presented with a situation that requires them to make a decision. The decisions will not be clear-cut, and therefore require arguments for and against different options. The examinees discuss and agree on a solution.
Example: (Task cards given to two examinees).
Your company has been invited to take part in a trade fair, which will take place during the busiest time of the year. You have been asked to decide whether staff should be sent to this trade fair. Discuss and decide together:
Advantages and disadvantages of attending the trade fair
Which members of staff would most usefully represent the company at the trade fair.

14. Tasks to Assess Speaking: Explaining and Predicting Task
Explaining and predicting tasks are designed the assess the examinee’s ability to explain things such as graphs and processes. The tasks will also require the examinee to describe the significance of the process or graph that they are describing. Example: (Examinee receives picture of a graph). 1. Tell the examiner about the information in the graph. 2. What do you think may be the reason for some of the changes in the graph?

15. Tasks to Assess Speaking: Structured Speaking Task
In a structured speaking task, the examiner asks the examinee a series of scripted and planned questions. The questions usually have specific answers so that the examiner knows what the examinee is going to say. Structured speaking tasks are designed mainly to test linguistic features such as pronunciation and grammar. These tasks may ask the students to read aloud or give short answers to questions. Example: The examinee is handed a booklet of prices for activities available on their vacation to Mexico. The examiner asks the examinee questions about the prices of various activities.

16. Tasks to Assess Speaking: Role-Play
Role-play tasks are completed as pair work. Each examinee is given a booklet with a list of the information that they need to tell their partner, and information that they need to get from their partner. The objective is to engage in fluent communication. An example of a role-play task is on the following slide.

17. Tasks to Assess Speaking: Role-Play
Example: Each examinee is assigned a role. In this case, one examinee is the babysitter and one examinee is the parent. An excerpt from a role-play task would look like this:
Examinee 1: Find out from the parent
What time the child should go to bed.
What time the parents expect to be home.
If you are expected to make something for dinner. If so, gain further information about the meal.
Examinee 2:
Tell the babysitter what time you’d like your child to go to bed.
Let the babysitter know what time you can be expected home.
Tell the baby sitter that you left an easy meal for the child, and how to prepare it.

18. Tasks to Assess Speaking: Interview
Interviews are completed individually. Each examinee is interviewed one- on-one with the examiner. The test-taker typically does not know what questions they will be asked. The objective is to check the test-taker’s ability to speak fluently across a range of topics. An example of an inteterview task is on the following slide.

19. Tasks to Assess Speaking: Interview
The Oral Proficiency Interview (OPI) is a test used in the United States and is designed to assess the examinee’s level of language proficiency. It takes place in a minute telephone call with an examiner. These interviews have 4 distinct parts:
Warm-up – greetings, informal exchanges, conversation openers
Level Checks – The interviewer initiates conversation on several different topics to determine the ‘performance floor’ of the examinee. This tells the interviewer what kinds of tasks the examinee can handle.
Probes – The interviewer continues the interview, probing to determine the performance ‘ceiling’ of the examinee. This is the highest level at which the examinee can speak before weaknesses in language use are exposed.
Wind down – a return to a comfortable level of speech

20. Tasks to Assess Speaking
When deciding which speaking tasks to use in assessment, remember to consider speaking definition construct, test purpose, and target language use. This will help you determine which speaking tasks to use for your assessment.

21. Reflect Choose a task that you want to try with your class:
-Description task Explaining and Predicting task
-Narrative task Comparing and Contrasting task
-Instruction task Structured Speaking task
-Decision task Role-play
-Interview
When choosing a task, consider the following questions:
How will you use it in a test? Who will the examinees be? How will you grade it?

22. Write Speaking Prompts

23. Speaking Prompts
A prompt is often used in productive tasks (when we want the examinee to speak or write). For example: Prompt: Is modern medicine or traditional medicine better? Give two reasons for your opinion. Directions: You and your partner will discuss whether traditional medicine or modern medicine is better. You have 2 minutes to prepare and 3 minutes to speak. Each person should think of two reasons for their type of medicine.

24. Speaking Prompts
The speaking prompt contains the question, or the task that the student needs to complete.
Write a question that allows students to demonstrate their mastery of specific skills.
When creating a prompt, you should be as clear and unambiguous. Excessive wordiness or an unclear question can increase the difficulty for students.
Make sure that the task does not require outside knowledge.

25. Remember
We have talked about test directions and speaking prompts. What can you remember about them? What elements should speaking prompt directions have? Write what you can remember. For example: Directions should contain information about how long examinees have to respond.

26. Remember: Feedback
Here are the elements that good directions should have: -Purpose of the task -Time limit -Approximate length of response -Directions for responding -How to record the answers -How responses will be scored/graded

27. Speaking Prompt Directions
It may seem like a lot of information to give examinees. Think about why each element is important for examinees to know. -Purpose of the task -Time limit -Approximate length of response -Directions for responding -How to record the answers -How responses will be scored/graded

28. Speaking Prompt Directions
Purpose of the task: Why is the examinee doing the task? What will the task tell you and the examinee about his or her abilities? Time limit: How long does the examinee have to do the task? Approximate length of response: Imagine an examinee who speaks for 30 seconds when you want a response that’s 2 minutes long.

29. Speaking Prompt Directions
Directions for responding: Does the examinee have time to prepare before responding? How to record the answers: Should the examinee audio record him or herself? If so, you should include instructions for how to do so. Will the response be graded live? If so, include any direction you have for the students. How responses will be scored/graded: Are responses graded for fluency? Accuracy? Pronunciation? Grammar?

30. Speaking Prompt Directions
Most importantly:
Directions should be simple and clear
The student should understand what they are being asked to do

31. Speaking Tests

32. Brainstorm
Make a list of things that you think make a good speaking test.

33. Brainstorm: Feedback
Here are some things that make a good speaking test: Validity Reliability Practicality We will discuss these aspects in the next section.

34. Speaking Tests Validity Reliability Practicality
Compared to listening or reading tests, rating speaking requires more resources in terms of personnel and time. A few concerns when assessing speaking are:
Validity
Reliability
Practicality

35. Speaking Tests: Validity
Validity is the extent to which a test measures what it is intended to measure. To have validity, the speaking task and the way in which we are rating the task must represent the construct and the target language use we are assessing.

36. Speaking Tests: Validity
How can we make speaking tests more valid? Before creating the test, make sure you have: 1. A clear construct definition 2. An explanation of target language use 3. An individual to review the test to make sure that there is a match between the construct definition and the target language use

37. Speaking Tests: Reliability
Reliability is the extent to which a test gives consistent results. There are two main aspects of reliability that raters must consider when rating tests: Inter-rater reliability: Do different raters agree with each other across all of the speech samples? Intra-rater reliability: Does one rater consistently rate the same samples at two different points in time?

38. Speaking Tests: Reliability
How do we increase reliability?
Have two raters instead of just one rater. Using two raters decreases potential rater bias.
Have rater training. Ensure that raters are familiar with the speaking task prompt and that they are familiar with the rubric.

39. Speaking Tests: Reliability
How do we increase reliability? 3. Read and discuss rubrics with other raters. Review language samples which have been previously rated by expert raters and discuss the ratings given. 4. Practice rating a different set of language samples. Then compare the ratings with those of experienced raters. Discuss the ratings and how the criteria were applied.

40. Speaking Tests: Practicality
Practicality is the relationship between the resources (time, money, people) required to design, give, and grade a test and the available resources.

41. Rating Speaking Tasks: Practicality
How do we ensure that a speaking test is practical? 1. Make an inventory of the resources that are available to you. 2. Make sure that it matches with the resources that you need to administer and score the test.

42. Reflect
Think back to tests you have administered to your students. Consider the following things:
What level were your students?
What speaking tasks did you use? (ex: narration task, decision task, role-play, interview)
After considering the aspects of speaking assessment presented in this lesson, do you think that your speaking test successfully measured your construct? Why or why not?

43. Key terms & Concepts Review Terms:
Validity Reliability Practicality

44. Next Steps
Now, you will read about rating rubrics. This is activity #4.

45. Sources
Luoma, Sari. (2004). Assessing Speaking. Cambridge: Cambridge University Press. Examples adapted from: