The Place of Direct and Indirect Tests in the Assessment of Speaking Skills: A Case Study Dr. Hale Kızılcık May 30, 2012

OUTLINE ENG 211 Academic Speaking Skills The development of the midterm exam A closer look at the exam: «Discussion skills» Exam results Interpreting the exam results

ENG 211 Academic Speaking Skills Revised in Not only to focus on «speaking as performance» (presentation) But also to focus on «speaking as transaction» (Richards, 2007)

Assessment PRESENTATIONS 50% Team Presentation Informative Presentation Final Presentation 10% 30% SPEAKING ACTIVITIES20% Current Events News Story Going One Step Further Tasks 5% 15% MIDTERM EXAM20% WRITING ASSIGNMENTS 10% Speech Analysis Report Self-evaluation Report 5%

Preparing a written test for a speaking course Valid Reliable Authentic Practical Backwash (Hughes, 1998)

Course objectives Cooperation with the course coordinators Face validity

Parts of the Midterm Exam Listening for specific information and main ideas (podcast) 6 pts. Synthesizing & expressing opinions (6 pts.) *Listening for note-taking Organizing a speech (4 pts.) Discussion skills (4 pts.)

Discussing testing «discussion skills»  «They already do it in the class (a lot of times!)»  «Is it a good idea to test spoken language in written discourse?» «It may have benefical backwash effect. It can encourage students to pay attention to signposts» «They already do it... But let’s see if they can integrate sources and use appropriate strategies» «In this way, we can also test their critical thinking skills»

Part IV: Discussion Skills (4 pts.) The following excerpts are taken from discussions. Participate in each discussion as instructed. Express your views in 3-4 sentences. (2 pts. each) A: Read the transcript of a discussion on graffiti. Interrupt the last speaker appropriately, agree with Victoria, and support your stance by giving your own reasons. Victoria: I think one should consider graffiti from a broader angle. Graffiti is not limited to painted large letters on walls. Think of a graffiti artist like Banksy. I would love to have one of his paintings on my wall. Sadi: I do not think graffiti can be justified on any grounds. It is a form of vandalism and personally I would sue anybody, Banksy or Picasso, who would paint graffiti on the walls of my house. You: ______________________________________________________________

Sample answer: I am sorry but I’d like to make a point here. I think Victoria is right and we have to distinguish between different forms of graffiti. Besides, I’d also really appreciate a Banksy on my wall. First, his graffiti is full of important political and social messages which are worth spreading. Second, they cost a fortune. One of his graffiti works can make me a rich man. Criteria for marking: Signpost and stance: 0.5 pt. Support: 1.5 pts.

Testing Discussion Skills Directly Sample speaking task & Debate & discussion rubric

Criteria Quality of arguments (5 pts.) Style (5 pts.) Delivery (5 pts) Language (3 pts.) Team work (2 pts.)

Interpreting the Midterm Results

Success Rate in Different Sections of the Exam Part I (listening): 61% Part II-A (synthesizing): 72% Part II-B: (expressing opinions): 77% Part III: (organizing a speech): 78% Part IV-A: (discussion skills): 49.5% Part IV-B: (discussion skills): 49.5%

CHART DISPLAYING ST. GRADES IN PART IV AND IN-CLASS DISCUSSIONS

So? What does this mean? Is there a problem with the reliability of in-class speaking tests? Is there grade inflation on the part of the teacher? Is there a problem with the discussion grading rubric? Is there a problem with the validity of the written test? Are the two tests testing different skills?

A closer look at the extreme cases Grammar becomes more important in the indirect test. Repetitions & digressions may be overlooked in the direct test. Quality of ideas are scrutinized more in the indirect test. The criteria for the direct test is broader in scope. There are personality issues.

Conclusions On testing speaking indirectly On testing speaking as performance On structures speaking tasks & automatic scoring of tests (Luoma, 2004) On teaching discussion skills

Thank you

REFERENCES Hughes, A. (1989). Testing for language teachers. Cambridge: Cambridge University Press. Luoma, S. (2004). Asessing speaking. Cambridge: Cambridge University Press. Richards, J., C. (2007). Developing Classroom Speaking Activities: From Theory to Practice: Summary article. From classroom-speaking-activities.pdf