A Customized Speaking Rubric: Design & Implementation Devrim Uygan & Eylem Mengi School of Languages/Sabanci University Istanbul/Turkey

Who are we? Sabanci University English Preparatory Program at SL 650 student intake 4 Routes (A1, A2, B1, B2) Speaking 10% (Assessed twice at 5%) A1 & A2 (Interlocuters) B1 & B2 (Group Discussion)

One of the products of a large scale research project by Spoken English Research Group:

OUTLINE The Birth: (What we did before creating the new criteria) First Baby Steps: How we actually made the criteria First interactions:Our Key Principles Baby going to Kindergarten: Getting Feedback Gaining independence:Training teachers, and Standardization Flying the nest- soon!

We identified/analyzed the needs of: The FDY students The FDY Instructors The faculty members

We did literature review and defined/identified: Key concepts Exit level descriptors Speaking objectives Main areas of development (Idea Dev., Interaction, Fluency&Coherence, Use of Language) & Subskills for each area Key principles for designing tasks and rubrics

Why a customized rubric? ! A productive skill- the speaking exam has the elements of progress, achievement and proficiency tests ( Hard to find a speaking rubric that caters for local needs) ! The assessment task types determine the specifications within the descriptors ! How much emphasis is given to speaking within the course/ TLP determine the expected outcomes and therefore the performances deployed in the descriptors.

There are some universal principles though…

The Old Criteria

The problems we had 1.The descriptors in each band were confusing a. Words were too general (e.g speaks clearly (A) //has a reasonable range of language to give clear descriptions (B)) b. Lacked parallellism between bands. c. 0-9 bands (full of lengthy descriptions with vague language)

2. Examiners could not differentiate between bands 3. Students could fit into different bands for different areas and scoring scheme did not allow this. 4. Areas overlap (e.g. Range is a part of all areas): Therefore, language could receive 75% of overall grade.

5. The Feedback sheet given to students was problematic

Our KEY Principles (How we formulated our rubrics) 1.Chose columns (Areas) that matched with our objectives a. Idea development//Task performance b. Interaction// Task performance C.Fluency and coherence d. Use of Language 2. Generic but local/route specific 3. Callibrated with CEFR descriptors (but also specific to our local condition)

3. Holistic but analytical enough 4. Reader-friendly language a. Parallel Language b. Avoided negative statements (Focus on what the student CAN do) c. Focus on performance (not speaking ability)

5. User-Friendly a. Easy to score b. No need to give additional feedback c. Score can be converted to a grade over 100. d. NO need to take detailed notes of student performances.

B1 Exit level Criteria…Activity

Watch a video

Developing the rubric together Collecting feedback, Revising Training, Standardization

Familiarizing Ts with the rubric in a training session. Providing Ts with a standardization video and detailed benchmarks & comments a week before the assessment. Grading another performance during a standardization session a day before the exam- with another video sample and benchmarks & comments readily available to the Ts.

Student B (In the middle)Benchmarks & Commentary Extracts from his performance (longer turns mostly- in chronological order): Extract 1. I think so because if you try to speak in lesson you won’t learn and do very well Extract 2. I don’t think so because we tried this while lessons… we study grammar… we tried to find similarity our languages but I think we shouldn’t do that… because we should look the English... or the other foreign language.. new language new horizon Extract 3. I think attendance is very important things too in our learning English...er...if we miss a lot of class we’ll fail the exams because exam plots are related of our course book totally similarities…er….we shouldn’t miss the classes Extract 4. I think so..I think our country’s education system of foreign language is so bad..I think you’re true is…er…not speaking on classes one or two times in one year we’re learning grammar everytime we learn grammar in high school maybe..native speakers of English…doesn’t know Extract 5. On the other hand, from the other side, if I were the director of school of language I would check tutorials because I think this is the most important thing cause we speak..we’re speaking with our tutorial.. with our tutors in our tutorials.. If there’s a problem on the tutorial I’ll try to check Idea Development (2): The student puts forward two new and relevant ideas and he contributes to the discussion especially with 2 longer turns, but this is not enough for band 3. He attempts to explain viewpoints and give specific information and reasons at times, but only some of these ideas are sufficiently developed. Interaction (3): He uses interaction patterns/communicative strategies three times: agrees and disagrees (e.g. I think so, I don’t think so); change the subject (on the other hand). However, he needs to interact with other students more often and use a variety of communicative strategies to be placed in band 4 from the criteria. Fluency and coherence (3): He attempts to produce linked speech and does contraction, which are good signs of continuity, however frequent pausing and hesitation, trying to search for words or form phrases, make the speech unnaturally slow at times and sometimes reduces fluency. Use of Language (3): He demonstrates limited range and control of lexis and structures at times i.e. prepositional phrases, which prevents full expression of ideas, yet none of these errors impede the message clarity and the student keeps going comprehensibly.(Except in extract 2, where lack of clarity seems to stem from insufficient coherence and idea development).