Workshop LSP Assessment Christian Krekeler Konstanz University of Applied Sciences krekeler@htwg-konstanz.de

LSP tasks for teaching LSP tasks for assessment 1 LSP tasks for teaching LSP tasks for assessment

LSP teaching, LSP tasks Task Renewable energy supply: Make a video of a news item on a catastrophe/problem with a source of renewable energy (water, wind, solar). Work in groups of three or four students.

LSP teaching, LSP tasks Complexity Complex tasks Meaning Focus on a communicative goal Authenticity Real-world relationship Cognition Congintively demanding tasks Skehan 1998; Ellis 2003; Van den Branden 2006

LSP teaching, LSP tasks Task Content Language Renewable energy Types Applications Efficiency … Language Presenting Pronunciation Genre Terminology Structures

LSP tasks and language assessment Task Renewable energy supply gone wrong: Make a short video of a news item on a catastrophe/problem with a source of renewable energy (water, wind, solar). Work in groups of three or four students. Why is this a problematic task for assessment purposes?

task design: adapting LSP tasks to assessment purposes 2 task design: adapting LSP tasks to assessment purposes

TBLT and Assessment mismatch LSP TBLT LSP Assessment Complexity Meaning However, more often than not, tasks set for the assessment do often not incluce these qualities. They are often very different and there is a mismatch between the course content and the assessment. This should be avoided because assessements communicate to learners what is valued. Authenticity Cognition

Task-based language assessment TBLA is a direct performance assessment which uses authentic, complex and cognitively demanding tasks. Such "holistic" assessments are scored with regard to the desired communicative outcome of the task. My first point is therefore that assessment and couse content should be alingned and that TBL courses should include TBLA which can be defined as: I will return to the scoring aspect later on in my talk. Ellis 2003; Wigglesworth 2008

TBLT and Assessment LSP TBLT LSP Assessment Complexity Complexity Meaning Meaning If a language course is based on the use of complex, meaningful, authentic and cognitively demanding tasks, this should be reflected in the assessment. This relationship works both ways, as the assessments will also impact the curriculum, causing positive washback. Authenticity Authenticity Cognition Cognition

LSP tasks and language assessment Content and Language Tensions Task Difficulty Generalizability From an assessment perspective, it is positive to have a good performance sample from a learner. However, the are also tensions: content and language: role of background knowledge, content as construct irrelevant variance task difficulty: complex tasks, task features generalizability: language use in other situations, rationale of performance testing scoring: communicative goal Scoring

LSP tasks and language assessment Assessment Task Renewable energy gone wrong: Make a short video of a news item on a catastrophe/problem with a source of renewable energy (water, wind, solar). Work in groups of three or four students. Alter/replace this task to make it more suitable for assessment purposes.

LSP tasks and language assessment Alter/replace this task to make it more suitable for assessment purposes. compare results describe competences reduce complexity control input

LSP tasks and language assessment Assessment Task The dam of a large reservoir used for hydroelectricity is leaking. Write and read a news item on the possible reasons, the efforts of the engineers to stop the leakage and the design of the reservoir. Input: Use the chart "Hydroelectric dam". Language … Time …

3 Scoring Lsp tasks

Scoring TBLA quantitative scoring general impression scoring Focus on meaning and a communicative goal quantitative scoring general impression scoring global/holistic scoring Before we turn to rating scales, it may be useful to look at four different scoring methods and consider their usefulness for scoring LSP assessment tasks. systematic/analytic scoring

Quantitative scoring Counting errors Error ratios  objective, safe  not suitable for scoring complex LSP tasks for the very insecure teacher, who rates exams with the lawyers in mind, knowing that a score based on an error ratio is easily justifiable.

General Impression Scoring No fixed criteria unsystematic, difficult to legitimize improve accuracy by comparisons flexible, open mind

Global/holistic scoring Systematic/analytic scoring Rating scales Rating scale with global descriptors Global score for performance Rating scales Sub-scales for different features Separate scores Weighted overall score

Rating scale development Write descriptors for a rating scale for the task on the reservoir dam.

Rating scale development Specific descriptors – not: Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that makes regular interaction with native speakers quite possible without strain for either party. […] CEFR for the overly confident teacher: I know a good picture when I see one. At least they are keeping an open mind, wich may be useful because there are a wide range of performances. It is also a flexible method which can take unforseen strenghts or weaknesses into account. However, from a measurement perspective, GIS is a no go.

Rating scale development a priori method rating scale developed by experts "armchair approach" a posteriori method descriptors derived from observations of performance behavioural rating scales theory-derived rating scales Taylor 2011

Rating scale development Write descriptors for a rating scale for the task on the reservoir dam. specific, based on task observable Would you use your descriptors for global/holistic scoring or systematic/analytic scoring?

1 tasks and assessment 2 task design 3 Scoring LSP tasks LSP Assessment 1 tasks and assessment 2 task design 3 Scoring LSP tasks