STANAG 6001 Speaking Test Observations; Lessons learned Tamar Shavlakadze BILC Professional Seminar Tbilisi, Georgia 1-5 October 2017

General Outline of the Presentation Speaking test format (the core of the test and tester stance) Global tasks and communicative functions (macro and micro functions) Evaluation (Fluency & Accuracy)

Who are the test takers? Resolute Support Mission; Promotion to higher ranks; International positions abroad; Courses abroad; Graduates of English language courses of the Language Training School of NDA; Graduates of Bachelors Programme of National Defence Academy; Applicants for the positions of teachers and translators.

General Test Format Bi-level (levels 1-2; 2-3) Length: Bi-level 1-2 20-25 minutes Bi-level 2-3 30-35 minutes Two examiners (interlocutor/rater) Guided interview Three mandatory phases

Core of the test and three Perspectives Level Checks Probes Three Perspectives The core Psychological Proves the examinee what she or he can do Proves the examinee what he or she cannot do Linguistic Checks for tasks and content which examinee performs with greatest accuracy Checks for tasks and content which examinee performs with least accuracy Evaluative Finds the examinee’s speaking level Finds level at which examinees can no longer speak accurately

Tester stance TTT (Teachers/testers) talking time must be at a minimum in both cases; Lessons as well as speaking tests should be contextualized; Instruction checking questions should be used; Teachers/testers should create friendly atmosphere; Teachers/testers should be in control of the lesson/exam; Pursue topics with follow-up questions.

Tester Stance Testers Should NOT: Correct examinees; Finish their sentence; Use body language; Feed vocabulary item; Go informal or start chatting with candidates.

Global Tasks and Communicative Functions Macro function – categories for the functional use of spoken and written discourse consisting of a sequence of sentences. Factually oriented talk Evaluative talk description explanation narration justification instruction prediction comparison decision Bygate (1987)

Micro functions are related to individual actions completed within a turn in an interaction. CEF groups them into 6 main categories: giving and asking for factual information, e.g. identifying, reporting, correcting, asking, answering; expressing and asking about attitudes, e.g. agreement/disagreement, knowledge/ignorance, ability, intentions, preference, emotions, apologies, sympathy. suasion, e.g. suggestions, requesting, warnings, advice, asking help, invitations, offers; socializing, e.g. attracting attention, addressing, greetings, introduction;; structuring discourse, e.g. opening, turn taking, closing, changing the theme; communication repair, e.g. signaling non-understanding, appealing for assistance, paraphrasing.

Global tasks and Communicative Functions NO PROFICIENCY Level 1 SURVIVAL Short conversation Ask and answer simple questions (present tense) Handle a simple transaction (RP) Short Narration in present Short description Level 2 FUNCTIONAL Narration ( Present, Past, future) Description (person, place, object) Report/state facts on a current event Instructions /directions Compare/contrast Information Gathering Task (in past tense) Deal with a familiar situation with a complication (RP) Level 3 PROFESSIONAL Supported opinions Hypothesize Discussing an abstract topic State and defend policy

MEMORIZED PROFICIENCY Holistic Assessment Scale Levels & Tasks Accuracy Statements Text produced NO PROFICIENCY No functional ability None or occasional isolated words Random words & phrases 0+ MEMORIZED PROFICIENCY Three level checks at level 1 Can ask questions or make statements only with memorized material and set expressions Most utterances are telegraphic Linking words and markers are omitted, confused, or distorted Communication is severely limited Faulty pronunciation, stress and intonation Memorized words and short phrases 1 SURVIVAL Short conversation Ask and answer simple questions Handle a simple transaction (RP) Short Narration Short Description Can create sentences Can begin, maintain, and close short conversations Can satisfy simple, predictable, personal and accommodation needs Can meet minimum courtesy, introduction, and identification requirements Communicate about routine tasks in the workplace Seldom speaks with natural fluency, and cannot produce continuous discourse, except with rehearsed material Can speak at the sentence level and may produce strings of two or more simple sentences joined by common linking words Frequent errors in pronunciation, vocabulary, and grammar often distort meaning May only use one tense or tend to avoid certain structures Speech is often characterized by hesitations, erratic word order, groping for words Ineffective reformation and self-corrections Discrete sentences 1+ SURVIVAL+ Three probes at Level 2 (RP) Can easily take part in short conversations by asking and answering simple questions Shows a very limited and inconsistent ability to handle longer conversations on concrete topics Discourse consists of strings of related sentences but not full paragraphs Can satisfy a few social demands and provide more than skeletal information when supplying biographical background Can readily ask for assistance; request information and clarification; and express satisfaction, dissatisfaction, and confirmation May hesitate and even have to change the subject because of lack of language resources Simple structures and basic grammatical relations are only somewhat controlled Time references may be used incorrectly Vocabulary may be imprecise except for the highest frequency utterances Frequent errors in pronunciation, vocabulary, or grammar may impede communication Delivery may be labored Strings of related sentences

2 FUNCTIONAL Past narration Description Report on a current event Instructions /directions Deal with a familiar situation with a complication RP IGT in Past tense Can narrate current, past and future activities Can describe people, places, and things Can create full paragraphs, minimally cohesive Can confidently handle most normal, casual conversations on concrete topics Can often elaborate in common daily communicative situations Can give complicated and detailed directions and make non-routine changes in travel and other arrangements Simple structures and basic grammatical relations are typically controlled, while more complex structures are used inaccurately or avoided Vocabulary use is appropriate for high-frequency utterances but unusual and imprecise at other times Errors in pronunciation, vocabulary, and grammar may sometimes distort meaning Generally speaks in a way that is appropriate to the situation, although command of the spoken language is not always firm Full paragraphs, minimally cohesive 2+ FUNCTIONAL+ Three probes at level 3 Can create a significant amount of discourse beyond the paragraph level Can use the language effectively to narrate, describe, state facts, compare and contrast, give detailed instructions and directions Can use the language with less ease and effectiveness to support opinion, clarify points, and answer objections Shows some linguistic limitations when dealing with unfamiliar subjects and situations Can generally elicit information and informed opinion Can often discuss abstract concepts but can rarely use abstract linguistic formulations successfully Errors in vocabulary and more complex structures and cohesive features sometimes interfere with efforts to elaborate on an argument or point of view Pronunciation errors may occasionally impede communication Speech is usually appropriate to the situation Discourse beyond the paragraph level 3 PROFESSIONAL Supported opinions Hypothesizing Discuss an abstract topic State and defend Policy Clarify points Produces extended discourse and conveys meaning correctly and effectively Use of structural devices is flexible and elaborate Speaks readily and in a way that is appropriate to the situation Without searching for words or phrases, can use the language clearly and relatively naturally to elaborate on concepts freely Can convey abstract concepts Can easily repair the conversation Pronunciation may be foreign Errors may occur in low frequency or highly complex structures characteristic of a formal style of speech Extended discourse

Fluency & Accuracy Two components of assessment Opposite ends of a continuum in which at extremes speech is seen as accurate and disfluent or inaccurate and fluent Two components of assessment Accuracy of structure and vocabulary Quality and speed of delivery

‘Error gravity’ – determines seriousness of errors - John buy a newspaper every day. - Every day a newspaper buy. (James,1998)

Definitions of fluency Fluency markers flow of smoothness rate of speech absence of excessive pausing absence of disturbing hesitation markers length of utterances connectedness

Central part of fluency Speaking rate Speech –pause relationships Frequency of dysfluency markers ( hesitations, repetitions, self-corrections) “Small words” and generic words

High frequency concrete Voc- eyeglasses; mast; etc. Simple structures Range of Vocabulary Grammar Structures Level 2 High frequency concrete Voc- eyeglasses; mast; etc. Simple structures Level 3 Low frequency concrete voc - spectacles High frequency abstract Voc -obligation; reduction; permanent; dilemma Complex grammatical structures Level 4 Low frequency concrete voc – monocle; mizzenmast ; camshaft Low frequency abstract- conundrum; scoff; abhor; rebuke. Low frequency grammatical structures

Bibliography Testing Second Language by Glenn Fulcher Assessing Speaking by Sari Luoma Common European Framework of Reference for Languages: Learning, teaching, assessment Standardization Agreement 6001 (5th addition ) LTS/ALTS material (Trisection of speaking and OPI checklist)

Thank you