Nükte Durhan METU, Northern Cyprus Campus, School of Foreign Languages (Ankara, 30 May 2012)

General look at socio-cognitive framework Cognitive validity (theory-based) and routine skills Needs analysis results of the Academic Speaking Skills (211) Course Description of the speaking tasks in the course and evaluation from cognitive validity aspects Context validity: considering its relevant aspects and the speaking tasks Areas that need work: scoring validity

Performance tests are characterised by the relationship of the test task to the world beyond the test. Ideally tasks are… direct and authentic - situationally authentic - interactionally authentic But a test can never fully represent reality.

We want candidates to… perform relevant language tasks adapt their speech to circumstances make decisions under time pressure implement them fluently make any necessary adjustments as unexpected problems arise

Test Taker Context Cognitive Response Scoring Validity Score/Grade Consequential Validity Criterion- Related Validity

Test-taker characteristics Context focused Cognitive focused (theory-based) Scoring focused Criterion focused (external) Consequentially focused

The abilities to be tested are demonstrated by the mental processing of the candidate (the cognitive dimension) The use of language in performing tasks is viewed as a social rather than a purely linguistic phenomenon (the socio- dimension) Test construct resides in interactions of underlying cognitive ability and context of use, i.e. socio-cognitive model

Executive processes Conceptualizer Pre verbal message Linguistic formulator Phonetic plan Articulator Overt speech Audition Speech comprehension (Weir,2005) Executive resources Language knowledge Grammatical Textual Functional Social Content knowledge Internal External

Routine Skills (Bygate, 1987) Informational Routines expository routines: involve factual information evaluative routines: involve drawing of conclusions, expression of reasoning; explanation, predictions, justifications, preferences and decisions

Routine Skills (Bygate, 1987) Interactional Routines : telephone conversations interviews meetings discussions decision making tasks

Routine Skills (Bygate, 1987) Interactional Routines : They are assisted by… interaction management skills involving content-focused agenda management and interaction-focused turn-taking Improvisational Skills brought into play when an interaction falters

Informative Speech (individual) (10%) Persuasive Speech (group) (20%) Final Presentation (individual) (30 %) No assessment of discourse and strategic competences

Participants want more of the following: More focus on spontaneous speech A variety of speaking activities More presentations with smaller percentages Different types of presentations Focus on fluency Interactive activities Creativity and individuality Speech critiques

Debate Role plays Discussions Impromptu speech Pronunciation activities Critical thinking activities Asking & answering questions

Reporting the teams observation of a stores merchandising strategies and store design Organizing and sharing points in the group Presenting findings Giving evidence from the observations Evaluating the effectiveness of the strategies

To inform and enlighten the audience about a given topic on the general theme of technology

Choosing a news item related to art, analyzing it and presenting the analysis to class Includes: justification of the selection of the item reaction to points and style evaluation of the ideas

Giving a balanced view of the selected topic (related to the five themes in the book) Includes: extensive literature research reaction to the presented material/discussion or evaluation of the topic

Debates Discussions Panel discussion Role plays Conflict resolution tasks Three different activities should be carried out in class throughout the semester.

Setting: task Purpose Response format Known criteria Weighting Order of items Time constraints Setting: administration Physical conditions Uniformity of administration Security (Weir,2005) Demands: task Linguistic Channel Discourse mode Text length Nature of information Topic familiarity Lexical range Structural range Functional range

1. Rubric Is the rubric accurate and accessible? 2. Purpose Is the purpose of the test made unequivocally clear for the candidate?

3. Weighting Are any weightings for different test components adequately justified? 4. Known criteria Are the criteria to be used in the marking of the test explicit for the candidates and the markers?

5. Time constraints: particularly important for speech as normal speech takes place under time pressure Is the timing for each task (e.g. preparation and completion) appropriate?

6. Discourse mode: Reciprocity conditions are concerned with the dimension of interpersonal interaction, the relation between speaker and listener. Are there reciprocal exchanges where speakers have to adjust vocabulary and message and take each others contributions into account?

7. Type of information Is the type of information appropriate for the target situation requirements of the students being tested? 8. Content knowledge Is the content sufficiently familiar to candidates so that they have sufficient existing schemata to enable them to deploy appropriate skills and strategies? (Urquhart & Weir, 1998)

8. Content knowledge every attempt should be made to allow background knowledge to facilitate performance rather than allowing its absence to inhibit performance (Alderson, 2000) 30

9. Linguistic knowledge Lexical range Are the lexical items in the test both in input text and required as output appropriate for the level of the candidates? Syntax Are the grammatical items in the test both in input text and required as output appropriate for the level of the candidates?

9. Linguistic knowledge Function: is a term used to describe the illocutionary force of what is said. Are the functions in the test both in input text and required as output appropriate for the level of the candidates? The variety of tasks with different functional purposes and increasing number of tasks improve the validity.

Rating Criteria / rating scale Rating procedures Rater selection Rater training Standardisation Moderation Rating conditions Statistical analysis