The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee

Outline The design and implementation of the questionnaire Some overall themes Comparing results by regions Implications

The timetable First meeting: Preliminary design: Web-based questionnaire (roll-out date): Data collection: Data preparation (for analysis): Analysis + preparation for conference: 24 – 25 January February March March – 15 April – 21 April April - 12 May 2004

Part 2: Questions for teacher trainers/educators Part 3: Questions for people who work, full time or part time, for language testing/examination boards or organisations that design school-external, regional or nationwide tests/examinations

Summary of responses 914 respondents to at least one part of the questionnaire –Part 1: 741 –Part 2: 709 –Part 3: respondents have provided contact details

Summary of responses Point to Note –Respondents did not necessarily answer every item in each section for which they submitted data. Responses for parts 2 & 3 are particularly low –Part 1: 522 – 614 responses per item –Part 2: 181 – 212 responses per item –Part 3: 198 – 228 responses per item

Where do the respondents work?

’Teachers only’: degree of formal training in assessment 1= none 2= short 3= medium 4= long

Correlation between assessment-related activities teachers are engaged in and absence of professional training 1Classroom-focussed assessment

Teachers’ perceived needs: do you need training in this area? 1Classroom-focussed assessment

Correlation between assessment-related activities teachers are engaged in and absence of professional training 2Purposes of assessment

Teachers’ perceived needs: do you need training in this area? 2Purposes of assessment

Correlation between assessment-related activities teachers are engaged in and absence of professional training 3Content and concepts

Teachers’ perceived needs: do you need training in this area? 3Content and concepts

Correlation between assessment-related activities teachers are engaged in and absence of professional training 4External tests and exams

Teachers’ perceived needs : do you need training in this area? 4External tests and exams

Tentative conclusions on teachers’ needs Teachers generally perceive a need for training across the board Areas of particular need: –ELP/other portfolios, preparing own tests, peer/self assessment, interpreting results and giving feedback –To award grades, to find out what needs to be taught, placing students, awarding certificates –Aspects of culture, integrated skills, establishing reliability and validity, statistics –Reviewing and writing items, statistics, defining criteria, rating, interviewing

Teacher trainers: what they train 1Classroom-based

Correlation between assessment-related activities teacher trainers give training in, and absence of their own professional training 1Classroom-based

Teacher trainers: what they train 2Purposes

Correlation between assessment-related activities teacher trainers give training in and absence of their own professional training 2Purposes

Teacher trainers: what they train 3Contents and concepts

Correlation between assessment-related activities teacher trainers give training in and absence of their own professional training 3Contents and concepts

Tentative conclusions from teacher trainer data Teacher trainers perceive their own needs similarly to teachers Much of what they train in is in areas in which they have no professional training themselves (generally 30%-50%) Areas of greatest need, where teacher training is not given, or given by people with little professional training, largely coincide with the areas of greatest need in teachers: – ELP/other portfolios, preparing own tests, peer/self assessment, interpreting results and giving feedback + informal continuous assessment –To award grades, to find out what needs to be taught, placement, awarding certificates –Aspects of culture, integrated skills, establishing reliability and validity, statistics + productive skills

Correlation between activities experts are engaged in and absence of professional training 1Item writing and rating

Experts’ perceived needs : do you need training in this area? 1Item writing and rating

Correlation between which activities experts are engaged in and absence of professional training. 2 Developing tests and assessment systems

Experts’ perceived needs : do you need training in this area? 2 Developing tests and assessment systems

Correlation between which activities experts are engaged in and absence of professional training 3CEF related

Experts’ perceived needs : do you need training in this area? 3CEF related

Tentative conclusions on experts’ needs Experts generally perceive need for training across the board Areas of particular need: –Item writing, reviewing items or tests, making decisions about composition of tests –Statistical analysis, setting pass marks, creating/maintaining item banks, doing validation research –Relating existing tests to the CEF, using CEF as basis for new test systems

Regions compared 1. Southern Europe –Portugal, Spain, Andorra, Italy, Malta, Greece, Turkey 2. Western Europe –Ireland, UK, France, Belgium, Netherlands, Germany, Austria, Switzerland 3. Northern Europe –Iceland, Norway, Sweden, Denmark, Finland 4. Eastern Europe –Estonia, Latvia, Lithuania, Poland, Russia, Czech Rep., Slovakia, Hungary, Bulgaria, Romania, Ukraine, Slovenia, Croatia, Serbia and Montenegro, Macedonia, Azerbaijan 5. non-European countries Questionnaire 1 / Teachers

Teachers’ previous training 2 = A little (1-2 days)

Teachers’ need for training - average number of activities, etc where need for assessment training was reported (out of max 25)

Teachers’ reported need for advanced training 1 (activities out of max 25)

Teachers’ reported need for advanced training 2 (activities out of max 25)

Statistically significant differences Teachers’ previous training no differences between regions Teachers’ need for advanced training: Western Europe < Eastern Europe Western Europe < Southern Europe Teachers’ need for basic training: no statistically significant differences small n size made it impossible to calculate differences between individual countries

Top 5 needs (teachers / regions) 1st2nd3rd4th5th South Portfolio 73% Interpretation 65% Statistics ext. 64% Continuo us 62% Criteria 61% West Reliability 55% Validity 54% Statistics 50% Portfolio 45% Self / peer 45% North Portfolio 70% Self / peer 63% Reliability 62% Validity 62% Statistics 61% East Portfolio 72% Reliability 72% Statistics 71% Validity 71% Interpretation 65%

Implications for training event(s) Common training events could be designed for both teachers and teacher trainers These should aim to cover the needs revealed here as pressing; however, it is likely that more ’core’ needs may exist for grass roots teachers (further investigation needed) A separate, specialist, training event would be necessary to cater for experts’ needs Regional events may be adapted to reflect different priorities

And finally.. Beneficial to belong to EALTA?