1. Romanian Speaking Evaluation Cpt. Corina Ispas Romania

2. Testing Speaking From practice to perfection?

3. Romanian Speaking Evaluation- short history 1998 – testing based on scripted scenario 1999 – training received from DLIELC, USA 1999-2004 – testing speaking in accordance with the training received (OPI) 2002 – beginning of the new testing project (UK, USA) 2004 – re-shaping of speaking evaluation to accommodate national needs; 2008 – re- analyzing of the requirements to accommodate + rating and evaluation

4. STRUCTURE OF THE TEST Warm up Core Wind down

5. Phases of the test Warm up - to put the examinee at ease - to give an initial idea of the examinee’s level Wind down - to have the examinee leave the test with a feeling of accomplishment

6. Core of the test Level checks - confirmation of the examinee’s ability to perform the requirements of the given level; Level probe - the examiner’s attempt to raise the level of the examinee’s language by increasing the linguistic demand on the examinee.

7. From Old to New - From good to better ?! Evaluation of national needs – large number of population –O–OPI – time-consuming (max.40 min) – difficulties in identifying certified examiners (elicitation/response chain, three way conversation…) –F–Face – validity –R–Reliability dependent on human element

8. Test specifications Purpose e.g. types of speaking being targeted Tasks e.g. test method

9. Special characteristics Materials/ stimuli e.g. input Length Topics Number/length of tasks Channel

10. Candidates i.e. number, characteristics Examiners i.e. role, characteristics, delivery speed

11. Tasks: Test method Materials/stimuli Input Length Topics Number/length of tasks Channel question/answer, role-play, information gathering verbal and/or visual 30 words basic needs such as meals, lodging, transportation, shopping; travel, family, job, everyday activities, hobbies and interests (based on types of speaking elicited) 13,5 minutes, 3 tasks + warm-up and wind down face to face

12. Candidates: Number Characteristics Examiners Role Characteristics Delivery speed Administration guidelines Criteria of assessment one at a time female/male, military/ civilian working in the military, adults, unknown to the examiners examiner and assessors female/male, non-native speaker, RoSE trained occasionally slower (when re-wording) see attached document in accordance with STANAG 6001 (2 nd edition) RoSE rating grid

13. Level 1 Purpose In typical everyday situations: 1. can begin, maintain and close short conversations by asking and answering short simple questions 2. can exchange greetings 3. can meet minimum courtesy, introduction and identification requirements 4. can elicit and provide predictable, skeletal biographical information 5. can communicate about simple routine tasks in work place 6. can ask for goods, services and assistance 7. can request for information and clarification 8. can express satisfaction, dissatisfaction and confirmation

14. Level 1 Short conversation (1, 3, 8,+/- 5) Examinee asks questions (1, 4, 7) Role play/situation (2, 3, 6, +/-7, 8)

15. Short conversation – example of level 1 task Do you have a family? Who are the members of your family? Do you help your (member of family) in the house? What are some of the activities you usually do? How often do you...(name the activity)? What is your favorite activity? When do you usually do this? - begin, maintain, close short conversation (1) - meet minimum courtesy, introduction and identification requirements(3) - express satisfaction, dissatisfaction and confirmation(8) - communicate about simple, routine workplace situation (+/- 5)

16. Short conversation 3 min. IG 3 min. Role Play 3 ½ min Warm up 2 min. Wind down 2 min.

17. Level 2 Purpose In typical everyday social and routine workplace situations: 1. can describe people, places and things 2. can narrate current, past and future activities 3. can state facts 4. can compare and contrast 5. can give straightforward instructions and directions 6. can ask and answer predictable questions 7. can often elaborate in common daily communicative situations

18. Level 2 Narration and description (1, 2,+/- 3) Compare and contrast (3, 4) Instructions/Directions (5) Role-play/situation (6, 7)

19. Narration + description – example of level 2 tasks People take many courses in the military. Think of one of the courses you have attended. Tell me what you did there on the first day. Describe one of Instructors so that I can recognize him/her. - can narrate current, past and future events (2) - can describe people, places and things (1) - can state facts (+/- 3)

20. Narration and description 3 ½ min Compare and Contrast 3 min Role Play 3 ½ min Warm up 2 min. Wind down 2 min. Instr./direction 1 ½ min

21. Level 1-2 Warm up 2 min. Short conversation 3 min. Narration in the past Description 3 ½ min IG 3 min. Compare/Contrast 3 min. Role Play 3 ½ min Instr./direction 1 ½ min Role Play 3 ½ min Wind down 2 min. Level 1 Level 2 Basic questions on surrounding objects 3 min. Level 0

22. Testing for level 1-2 – example of variant Warm up Level 1 Level 2 Short conversation- 2Narration + description- 7 Probe L2 (Narration/ description) -7 IG- 5Compare + contrast - 15 Role Play- 11Instructions/ Directions- 9 Role Play- 4 Wind down

23. In most formal and informal conversations on practical, social and professional topics: 1. can discuss particular interests and special fields of competence 2. can use the language to perform such common professional tasks as answering objections, clarifying points, justifying decisions 3. can demonstrate language competence when conducting meetings, delivering briefings, or other extended and elaborate monologues, hypothesizing and dealing with unfamiliar subjects and situations 4. can reliably elicit information and informed opinion 5. can convey abstract concepts in discussions on different topics 6. can produce extended discourse and convey meaning Level 3

24. Level 3 Supported opinion with descriptive prelude (1, 2, 5, 6) Regular monologue (2, 3, 6) Role play/situations (2, 4) Hypothetical monologue (2, 3, 6)

25. Supported opinion with descriptive prelude – example of level 3 task Leadership is one of the most attractive and difficult to define of all human qualities. On The basis of your experience, what do you think makes a leader and which are the qualities and characteristics most frequently associated with leadership? Give 2-3 reasons to support your opinion. - discuss particular interests and special field of competence (1) - justify decision (2) - convey abstract concepts in discussions on different topics(5) - can produce extended discourse and convey meaning (6)

26. Supported opinion 4 ½ min Regular monologue 3 ½ min Hypothetical monologue 3 ½ min Warm up 2 min. Wind down 2 min. Role Play 5 ½ min

27. Level 2-3 Warm up 2 min. Narration in the past Description 3 ½ min Support opinion 4 ½ min. Compare/Contrast 3 min. Reg. monologue 3 min. Role Play 3 ½ min Role Play 5 ½ min Hypo. monologue. 3 ½ min Wind down 2 min. Instr./direction 1 ½ min IG 3 min. Role Play 3 ½ min Short conversation 3 min. Level 1Level 2Level 3

28. Testing for level 2-3 – example of variant Level 1Level 2Level 3 Short conversation - 2 Narration + Description - 7 Supported Opinion - 4 Probe L2 (Narration + Description) - 7 Probe L3 (Support Op.6)Regular Monologue - 9 Probe L3(Reg. Monologue 9) IG- 5Compare + contrast 4Role Play - 14 Role Play - 11Instructions/ directions 4Hypothetical Monologue - 7 Role Play 4

29. RATING Scale focusing on 5 areas of “interest”: Task Fulfilment Lexical & Structural Control Delivery Output Communication Strategies

30. TASK FULFILMENT LEVEL 1 Can communicate in simple everyday situations. Functions are typically satisfied within task parameters. Response to interlocutor/situation is sufficiently appropriate. LEVEL 2 Can communicate in everyday social and routine workplace situations. Functions are clearly performed within task parameters. Response to interlocutor/situation is generally appropriate. LEVEL 3 Can communicate in most formal and informal conversations on practical, social and professional topics. Functions are effectively performed within task parameters Response to audience/situation is on the whole stylistically appropriate.

31. LEXICAL AND STRUCTURAL CONTROL LEVEL 1 Frequent errors in vocabulary and grammar. Time concepts are vague. May use only one tense or tend to avoid certain structures. Vocabulary is limited to basic needs and daily routine. LEVEL 2 Basic grammatical structures are typically controlled. Uses concrete vocabulary. Grammatical and/or vocabulary errors may occur. Although they may cause misunderstandings, the message is conveyed. LEVEL 3 Effectively combines structure and vocabulary to convey meaning accurately. Structural inaccuracy is rarely the cause of misunderstanding. The use of structural devices is flexible and elaborate. Able to use low frequency vocabulary to convey abstract topics. Errors may occur in low frequency vocabulary and some highly complex structures.

32. DELIVERY LEVEL 1 Seldom speaks with natural fluency. Pronunciation, stress and intonation often distort meaning. Speech is often characterized by hesitations, erratic word order, frequent pauses, straining and groping for words. LEVEL 2 Handles the tasks with confidence but not with facility. E.g.: pauses for lexical and grammatical planning. Pronunciation, stress and intonation may distort meaning. LEVEL 3 Speaks readily and naturally in a way that is appropriate to the situation. No need to search for words and phrases. Errors in pronunciation do not distort meaning. Pronunciation may be obviously foreign.

33. OUTPUT LEVEL 1 Can speak at the sentence level and may produce strings of to or more simple, short sentences joined by common linking words. LEVEL 2 Organizes speech in complete but simple paragraphs. LEVEL 3 Produces extended discourse.

34. ADMINISTRATION AND RATING ISSUES - administration guide lines - two examiners - one examinee - interaction - rating process - monitoring

35. Measures Intensive training - refresher annual training (40 hours; 5-10 practice interviews) - annual recertification required - random observations of the evaluation process Task development process - task developer’s network - “to…from” feedback process - field-testing of the newly created items

36. Conclusions Questions ??? Thank you !