Weaving Pathways: Interculturalism and Language TESL Canada Conference 2009 Weaving Pathways: Interculturalism and Language

ELTPA Assessment Tool for Placement in ELT Programs Audrey: Welcome introductions, and agenda Materials needed: Laptop, LCD, Speakers Test Kit + video of Speaking Assessment (Zeeshan or Haider) Carolyn Cohen has worked in the field of ESL for 20 years. She began as an ESL Teacher and Teacher Trainer. She worked as a TSE evaluator and LINC Assessor and became fully engaged in LINC and ESL as a programs and services manager. She managed development of the CLBA/LA, the ELTPA, the CLBA International and the original form of the CLBPT and CLB Online self-assessment in partnership with the CCLB. Carolyn is Senior Project Advisor for the Centre for Education & Training where her work areas include the National ELTPA Support Services, CLB On-Line Self Assessment, and Language Assessment Support Services. In November 2008, Carolyn was recognized by TESL Ontario as one who has demonstrated significant ongoing commitment to our profession. Audrey Bennett is the Project Manager for the National ELTPA Support Services. She has worked at the Centre for Education & Training since 1992 in assessment, coordination of LINC Assessment Centres, training assessors in the use of CLB-based assessment tools, including CLBA, CLABLA, CLBA-International and ELTPA. presented by: Centre for Education and Training Carolyn Cohen, Sr. Program, Advisor Audrey Bennett, Project Manager

Centre for Education & Training Test Development CLBA Placement CLBLA Literacy CLBPT Streamlined CLBA –I High stakes CLB-OSA (English) Online Self- NCLC-AEL (French) Assessment ELTPA (6-10) Placement Carolyn NCLC-AEL (French) = Niveaux de Compétences Linguistiques Canadiens (NCLC) – Autoévaluation en ligne 3

ELTPA (6-10) Assessment referenced to CLB 2000 identifies proficiencies from Bm 6- through Bm 10+ for assessment of clients with prior CLB scored at Bm 6 or higher designed for placement in ELT programs workforce-oriented tasks and themes four language skills and four scores time-efficient, easy to use and reliable administered and scored by a trained assessor Funding: ELT Initiative and National Settlement and Resettlement Carolyn 4

Key Strategies consistency in application of development methodology consideration for stakeholder issues ongoing consultation with experts in the field rigorous validation process Carolyn: formulating development methodology and meticulous application stakeholders: the ELT initiatives and its goals the ELT providers and their needs language assessors and administration constraints consultants: test development CLB experts university-based ESL experts workforce training experts consistent application of validation methodology

Successes statistical analysis of pilot data confirms the validity of the speaking, listening, reading and writing sub-tests, and reveals reliability results commensurate with or exceeding standards for low-stakes tools positive feedback from candidates, assessors, and ELT instructors and administrators Carolyn

Recommended Uses for ELTPA Instruments determine client proficiency for placement in ELT programs provide instructors and service providers with accurate information on the clients’ proficiency facilitate program planning and development provide candidates with a better understanding of their higher-level workforce-context language skills Audrey Ontario region has provided ELT service Providers and Assessment Centres with a protocol for use of the ELTPA to place candidates in higher level language training programs.

ELTPA (6-10) Speaking and Listening 15 min; interview Task 1 and 2: introduce oneself and describe/narrate an experience Task 3: conduct an analysis Task 4: deliver a presentation Task 5: participate in a discussion Listening 20 min; assessor facilitated Task 1: follow a phone message Task 2: listen to a semi-formal presentation Task 3: listen to an informal conversation Task 4: follow a formal discussion Audrey

ELTPA (6-10) Reading and Writing 60 min; multiple choice Task 1: read an instructional email Task 2: read a company policy excerpt Task 3: read and compare ideas in two opinion passages Writing 45 minutes Task 1: write a memo/report Task 2: write a formal letter Audrey Reading Task 1 – message to someone replacing the writer while on vacation Task 2 – formal text outlining a company policy – contains acronyms referenced to the text Task 3 – editorial and response Writing Task 1 – regarding a workplace/professional challenge Task 2 – regarding a personal request and offering congratulation/support

2008-9 Implementation Project Outputs: Trained 45 new ELTPA assessors and 2 trainers (BC and Manitoba). This is in addition to Ontario (trained under separate agreement). Audrey To date there are nearly 100 trained ELTPA Assessors across Canada – federally funded (CIC)

Project Outputs – cont’d: Provided Information Sessions to ELT providers, CIC, community partners - Questions and discussion points from these sessions have led to development of a series of FAQs that will be reviewed by a Discussion Forum as part of the current agreement Audrey These sessions were well attended in most jurisdictions and attendees were invited to look at the assessment (including a video of the Speaking assessment), ask questions related to the assessment, their programs, and how ELT programs meet the needs of the community. Programs were given the opportunity to network with each other, the assessment centres, trainers and CIC and community representatives.

Project Outputs – cont’d Research Study (2009) Purpose: Comparison of standardized CLB Assessment results with ELTPA To address fact that some ELT candidates lack prior CLB-referenced scores (lengthens qualifying process) To address ELT program eligibility criteria for those lower that CLB 7 Carolyn Can the ELTPA Speaking Assessment be used as a screener to reduce the number of clients who have to take two complete language assessments? Can we find a way to predict, based on ELTPA Speaking, whether a client is more likely to achieve a meaningful reading, Writing and Listening outcome on CLBA/PT or ELTPA?

Project Outputs cont’d: Comparison Study To compare ELTPA results with those of CLBA and CLBPT where skills have been evaluated through prior CLB-referenced assessments to be BM 6 or higher - 152 case samples assessed on ELTPA and CLBA/PT - range of 29 occupations - trade certificate to doctoral degree Carolyn

Comparison Study (cont’d) ELTPA results are consistent with achievement of CLBA and CLBPT Benchmarks 6-10. Achievement of Benchmarks 9 and 10 cannot be measured using CLBA/PT

Potential Screening to ELTPA Can the ELTPA Speaking be used as a screening tool to reduce the number of clients who have to take two complete assessments? Could it be a predictor of outcomes in other skills?  Can we identify Benchmarks 5 and 6 for some ELTPA results? Carolyn

92% achieved Bm 7 or more in Listening Potential Screening to ELTPA (Continued) Sample of 241 cases 92% achieved Bm 7 or more in Listening 96% achieved Bm 7 or more in Reading Writing less promising – 54%

Recommendations Possible use of ELTPA Speaking as a screener where 2 or more Benchmarks at 7 are sufficient Expand ELTPA Speaking component to identify Benchmarks 5 and 6 mean fewer candidates require two assessments Further validation and consultation with Service Providers Carolyn

National ELTPA Support Services 2009-2011 Create a National Discussion Forum to address regional needs, issues, challenges and solutions for use of ELTPA for placement of learners in ELT programs Train up to 15 new ELTPA assessors per year in the administration of Enhanced Language Training Placement Assessment (ELTPA) tool to newly identified assessors Offer 1-day recalibration for current ELTPA assessors to maintain consistent scoring practices and CLB results determination Conduct an inter-rater reliability study on inter-scorer accuracy in administration and scoring of ELTPA to ensure consistency in scoring Consultations to examine when it is best to use the ELTPA in comparison to other assessments Audrey Discussion Forum – to start will have 11 members representing CIC, SPOs, assessment and training Training – first training under this agreement will be in October (22-23) Recalibration – anticipated start of this component – November Inter-rater reliability study – next fiscal year Consultations – to extend from the National Discussion Forum to promote uniformity of use and access for individuals seeking higher level language training