Critical Review: Chun, C. W. (2006). An Analysis of a Language Test for Employment: The Authenticity of the PhonePass Test Tianfei (Faye) Jiang & Peipei (Christina) Dong CTL 3799 Communication and L2 Learning in Workplace Prof. Julie Kerekes February 11, 2014

OUTLINE Background & Context Characteristics of the PhonePass test – Procedure – Setting Frameworks Analysis of the PhonePass test Conclusion Critique & Implications Questions & Comments

BACKGROUND Global technology has quickly progressed to provide more accessible business modes of telecommunication, like telephone, video phone, s and other internet/computer- mediated system in workplace. Tests over telecommunication networks are usually highly commercialized..

CONTEXT PhonePass Spoken English Test-10 (SET-10) WHAT IS IT? An expedient assessment tool for screening job candidates ability in spoken English WHAT IS IT FOR? Evaluating employees for promotions and measuring service employees with an accurate and objective test of their spoken English spoken capabilities. HOW DOES IT WORK? This test is a 10 minutes spoken English test for non- native English speakers, which is administered over a language-line telephone and immediately scored by computer. It consists of 5 sections, based on four diagnostic subscores which includes sentence mastery, vocabulary, fluency, and pronunciation. The cost ranges from $25 to $40. WHEN & WHERE IS IT? Anytime and Anywhere with a land-line phone system available. HOW IS IT SCORED? The first four sections are recorded and scored by computers, while the last section is not scored, which is open for review.

CONTEXT Purpose: to see if the characteristics of the domain of the test tasks found in the PhonePass SET 10 test correspond to real life domains of nontest tasks. Key Words: telecommunication; spoken language test; authenticity;

CHARATERISTICS IN PHONEPASS TEST Access to an Internet- connected Computer Download and Print out a Test Paper Access to a Land-line Telephone Follow Further Instruction in the Phone Testing PROCEDURE (requires employees computer technology & telephone technology capabilities)

CHARATERISTICS OF PHONEPASS TEST SETTING Please read No. 3. The winter in Toronto is freezing. A: Reading Aloud War broke out. There are three basic ways in which a story might be told to someone. B: Repeating Sentence What season comes before spring? Does a blanket go on the bed or the wall? C: Giving Short Answers to Question in/ bed/ stay we wondered/ would fit in here/ whether the new piano D: Building Sentences Which would you prefer to live, in the big city or a small town? And why? E: Responding to Open Questions

CHARATERISTICS OF PHONEPASS TEST Grammatical knowledge Textual cohesion Sentence mastery Vocabulary Fluency Pronunciation No reciprocal tasks visual and aural channels Any location & time High degree of speed 10 minutes Flexibility & Short Length Input Expectations Tasks

FRAMEWORK Authenticity the degree of correspondence of the characteristics of a given language test task to the features of a target language use task. (Bachman & Palmer, 1996, p.23) the developers need to show that the content of the test tasks is representative of the demands of the corresponding task outside the test situation, and that the scoring reflects this. (Luoma, 2004) authenticity of input, purpose, and outcome (Lewkowicz, 2000) Test Fairness Framework access, administration, validity, absence of bias, and social consequence. (Kunnan, 2004)

ANALYSIS: Authenticity RationalePhonePass Test (PART A) Created real-life discourseStructured speaking task Grammatical KnowledgeWithout correct sentence production and comprehension Textual KnowledgeRead in random order of items Functional and Sociolinguistic Knowledge Standard pronunciation

ANALYSIS: Authenticity RationalePhonePass Test (PART B) Proof of comprehensionSounds reorganization and repetition & Memory Speech productionWithout discourse creation and interpretation

ANALYSIS: Authenticity RationalePhonePass Test (PART C) Vocabulary of professional employment level low level of lexical knowledge Extended production responsesLimited production responses Sociolinguistic knowledgeN/A

ANALYSIS: Authenticity RationalePhonePass Test (PART D) Fluency (constructing, reading and repeating the sentences) Inadequate constructing Pragmatic knowledgeN/A Sentence masteryOne in six chance of guessing word groups

ANALYSIS: Authenticity RationalePhonePass Test (PART E) AuthenticityExtended production responses Not scored

ANALYSIS : Fairness RationalePhonePass Test (PART A) Minimal biasNo biased content or language Educational access (speaking & listening) Advantages for test takers who have had more opportunities to learn and become familiar with the tasks on the test Telecommunication access (land-line telephone, printer & internet connected computer) The absence of supporting telecommunication in underdeveloped countries Appropriate accommodationsUncertain physical (e.g. sight and hearing) impairment Rescoring and reevaluationN/A

CONCLUSIONS The performance on these tasks does not necessarily predict or reflect the speaking ability of the test taker to function in real-life domain of workplace. PhonePass SET 10 is considered relatively less authentic. The test tasks are relatively less specific to a particular domain and are not interesting and engaging.

CRITIQUE Strengths: The author used strong theoretical basis to support his analysis. It provides a valuable evaluation as a reference, emphasizing the importance of authenticity, to language assessment in workplace. Weakness: This article does not offer further practical implication in telecommunication in workplace. It would be better if the author provides more constructive suggestions to the phone language test.

IMPLICATIONS Since telecommunication lacks non-verbal compensation, such as body languages and facial gestures, the requirement of language is much higher than daily face-to-face communication. This analysis brings readers to have more deep provoking thoughts about telecommunication networks, for example, how can we communicate effectively without these compensation, and how can we minimize the misunderstandings in telecommunication. For employers, phone interview or language assessment via telephone should focus more on the authentic communicative competence than on linguistic performance. Communication effectiveness is a critically important criteria in assessing individual's language ability in workplace. Standards for communicative language capability in workplace is needed to be further explored.

Questions AND/OR Comments?

----- Tianfei Jiang & Peipei Dong February 11, 2014