The International Legal English Certificate Issues in Developing a Test of English for Specific Purposes David Thighe, Cambridge ESOL EALTA Conference Krakow, May 2006

2 Outline  What is ILEC?  Features of ESP tests  Continuum of Specificity  Evidence from the Development Practice  Evidence from the Trialling stages  Future Directions

3 What is ILEC?  A test for law students & newly qualified lawyers  A test that assesses language skills in a legal context  Set at CEFR levels B2 & C1  Recognised by leading associations of Lawyers

4 …. One in which test content and methods are derived from an analysis of a specific purpose target language use situation (TLU), so that test tasks ….are authentically representative of tasks in the target situation, allowing for an interaction between the test taker’s language ability and specific purpose content knowledge, on one hand, and the test tasks on the other. Douglas (2000: 19) A definition of specific purpose assessment

5 Revisiting Douglas’ definition of LSP tests  All tests can be seen as lying on a specificity continuum, between the highly specific and the general purpose.  This continuum is multi-componential and includes the twin aspects of authenticity – situational and interactional. O’Sullivan (2006)

6 Specificity – a continuum PET, FCE, CAE etc BULATS BEC ILEC? Generalisability Increase in test of content knowledge A test of English for air-traffic controllers

7 Where does ILEC go on this continuum?  Test purpose  Needs Analysis/Viability Study  Development practice  Working together with subject experts at the planning, design and development stages  Evidence from Candidate Performance  the trialling stage on authenticity and specificity CLAIM JUSTIFICATION OF CLAIM SUBSTANTIATING CLAIM

8 Test Purpose To assess candidate’s ability in English in an international legal environment To allow candidates to demonstrate to employers their ability to communicate in English with clients and colleagues To allow candidates to demonstrate to educational institutions their ability to follow a course of legal study in English

9 Cambridge ESOL External content experts Item Writers and Chair Development practice Cambridge ESOL and External Content Experts Sources Tests for vetting Test construction Items produced and vetted

10 Development Practice Test Content  Reading, Listening, Writing and Speaking components  Tasks based on law-related texts and topics  Set in the context of International commercial law

11 Evidence from ILEC Trial Specific Issues  Authenticity of components  Specificity of ILEC components  Reliability issues  Practical constraints e.g. time, compulsory/optional parts, length of paper etc

12 Sample and Instruments  320 candidates from 12 centres worldwide  Candidates completed  ILEC Reading, Writing and Listening papers  General Placement Test  Information Sheets  Questionnaires  Instructors questionnaires

13 Trial Findings: Authenticity 1 strongly disagree0.5 2 disagree5.9 3 no opinion agree strongly agree26 total100 Table 6 Q7 Candidate Questionnaire: On the Reading paper the topics are authentic (I or my colleagues may be expected to cover these topics in our legal profession). Table 7. Q8 Candidate Questionnaire: On the Reading paper the texts and language are authentic ( I or my colleagues may be expected to read similar texts in our legal profession). 1 strongly disagree1.5 2 disagree7.3 3 no opinion agree strongly agree31.2 total100 Similar results for other papers Similar results for instructors

14 Trial Findings: Specificity Candidate Performance Table of Comparison of scores of candidates with or without professional experience in GPT, Reading, Listening and Writing Without professional experience With professional experience GPT Reading Listening Writing

15 Trial Findings: Reliability Cronbach’s Alpha = 0.81 to 0.89

16 Trial Findings: Time Constraints  Candidate Questionnaire Approx 45% responded that they strongly agree or agree that there is enough time, 33% strongly disagree or disagree, 22% indicated no opinion  Shadowing of candidates taking Reading and Writing components All candidates shadowed completed components, 30% writing candidates used full time allowed. 70% candidates completed tests in order of sequence in paper

17 ILEC Writing – Compulsory or Optional Part 2 Table: comparison of mean scores in both versions of writing trial paper  The ILEC Writing Test was offered to a group of trial candidates as two versions.  Version 1: Both parts compulsory (56 cands)  Version 2: Part 2 gives a choice of 3 options (48 cands)

18 ILEC: Future Directions  Evidence from Live Data on specificity and the role of content knowledge in the test  Performance of those with and without work experience  Using expert judgement to provide further evidence of Task Specificity

19 In Summary  Outline of ILEC  Defining features of ESP tests  Purpose from Viability Study  Test Development Process with External Expert  Empirical Evidence from ILEC Trial  Future Research

20 References  Douglas, D Assessing Languages for Specific Purposes. Cambridge: Cambridge University Press.  O’Sullivan, B Issues in Testing Business English: the revision of the Cambridge Business English Certificates SILT Volume 17. Cambridge: Cambridge University Press.  Weir, C Language Testing and Validation: an Evidence-based Approach. Hampshire: Macmillan