ASSESSING WRITING Prepared by Olga Simonova, Maria Verbitskaya, Elena Solovova, Inna Chmykh Based on material by Anthony Green

Outline of the lecture 1.Why teach writing? 2.Stages of teaching writing. 3.Writing: process vs. product. 4.Why test writing? Types of tests & their purpose for different test users. 5.Writing test specifications. 6.Rating scales in assessing writing skills.

WHY TEACH WRITING?

Why teach writing? 1.Children (school-based writing): –writing as a means of learning; –writing as a process; –writing as a product. 2.Adults: –socio-cultural experience (travelling, migration, etc.); –academic experience (furthering one’s education); –professional experience (career development).

STAGES OF TEACHING WRITING

6 Output Phonological/ orthographic encoding Grammatical and lexical encoding Planning and organisation Mental representation Productive language processing

Stages of learning and teaching writing: graphics (level of letters/ their combinations); spelling (word level); note-taking (words/chunks/sentences/plan, etc.); writing as a communicative skill.

Micro skills: produce graphemes and orthographic patterns produce writing at an efficient speed to suit the purpose produce an acceptable core of words and use appropriate word-order patterns use an appropriate grammar system (tense, agreement, pluralisation, etc.) express a particular meaning in different grammar forms. use cohesive devices in written discourse. (Brown 2003, p.221)

use the rhetorical form and conventions of written discourse; appropriately accomplish the communicative functions of written texts according to form and purpose; convey links and connections between events; communicate the main idea and supporting ideas; communicate given information and the new information, generalization and exemplification; Macro skills:

distinguish between literal and implied meanings when writing; correctly convey culturally specific references in the context of the written text; develop and use a battery of writing strategies such as: –assessing the audience’s interpretation; –using pre-writing devices; –writing the first draft: –using paraphrases and synonyms; –seeking peer and instructor’s feedback; –using feedback for revising and editing. Macro skills:

Types of tasks  Copying  Dictations  Gap filling (word/ word combination/ sentence level)  Scrambled words/ sentences  Sequencing/ ordering  Find and correct the mistakes  Short answer and sentence completion  Multiple choice questions  Multiple matching  Paraphrasing  Grammatical transformations  Vocabulary exercises  Cloze  Picture-cued tasks

WRITING AS PROCESS AND PRODUCT

Writing: Process vs Process Process: drill / controlled / guided ex-s; brainstorming, planning, all draft materials. Product: different types of written texts within different types of language course; letters / s / questionnaires, transactional letters; memos / minutes / business plan; essays, reports, articles, presentations, etc.

Genres and text types 1.Academic writing: general subject reports, essays, compositions, term/ course papers, academically focused journals, short-answer test responses; technical reports (lab reports); theses, dissertation, etc. 2.Job-related writing: messages/phone messages; letters/ s/ transactional letters; memos; minutes; reports (job evaluation, project reports); schedules, labels, signs; announcements, advertisements; manuals, etc. 3.Personal writing: letters, s, greeting, cards, invitations; messages, notes, reminders, shopping lists; financial documents (checks, tax forms, loan applications); forms, questionnaires, medical reports, immigration documents; diaries, personal journals; fiction, etc. (Brown 2003, p.221)

WHY ASSESS WRITING?

ConstructPurpose Task typesScoring criteria Assessing writing: key questions How (score)? How (score)? Why assess writing? Why assess writing? How (test)? How (test)? What is writing? What is writing?

Why assess writing? No single answer: different groups of language learners have different needs, such as: –international travellers: language for travel, leisure; –migrants: survival skills, access to employment; - students: notes, essays, dissertations; - professionals: s, reports, letters, memos. different users have different purposes when they seek information from tests; but most users of language do need to write.

1.SOCIAL NEEDS: international travellers: language for travel, leisure; migrants: survival skills, access to employment; professionals: s, reports, letters, memos. 2.ACADEMIC NEEDS: students: mobility, exchange programs, notes, essays, course work, dissertations, national exams. 3.EDUCATIONAL NEEDS: teachers: monitoring, control, assessment; learners: progress check, skills development. 18 Why assess writing?

Areas of Language Knowledge Organisational competence Grammatical competence vocabulary syntax graphology Textual competence cohesion rhetorical organisation Pragmatic competence Functional competence ideational functions manipulative functions heuristic functions imaginative functions Sociolinguistic competence dialects/ varieties registers natural or idiomatic expressions cultural refs & figures of speech Bachman and Palmer (2010) Language Assessment in Practice

Performance testing in writing:  samples of written language that simulate real-life behaviour (are authentic in situation and interaction);  complex events involving unpredictable variables;  observed & evaluated by agreed judging process, where facets of measurement interact to produce score. McNamara (1996)

WRITING TEST SPECIFICATIONS

ConstructPurpose Task typesScoring criteria Assessing writing: key questions How (score)? How (score)? Why assess writing? Why assess writing? How (test)? How (test)? What is writing? What is writing?

Writing test specifications: topic text type length areas to be assessed timing weighting pass level. (Coombe, 2011)

Parts of writing test tasks: rubric prompt / input expected response post-task evaluation.

Rubric: instructions and criteria The instructions and the set of criteria for evaluation of the writing task. It is supposed to: specify a rhetorical pattern, length, time; indicate the resources available for students (dictionaries, spell checker, etc.) and the delivery method (paper and pencil, etc.); indicate whether a draft or an outline is required; include the overall weighting of the writing task in the exam. (Coombe, 2010)

Writing prompt ‘the stimulus the students must respond to’ Hyland (2003 p.221) 3 types of prompt patterns: base prompt (see the handouts); framed prompt (see the handouts); text-based prompt (see the handouts). Kroll and Reid (1994 p. 233) 26

Expected response and relationship between input and response: length purpose text type functions linguistic features interactiveness reactivity: degree of reciprocity; scope: amount of input needed to respond; directness: direct / indirect correlation / real world knowledge involved apart from the input. (Bachman & Palmer, 1996)

RATING SCALES

ConstructPurpose Task typesScoring criteria Assessing writing: Key questions How (score)? How (score)? Why assess writing? Why assess writing? How (test)? How (test)? What is writing? What is writing?

Task analysis IELTS Academic Writing Task 2: Upbringing WRITING TASK 2 You should spend about 40 minutes on this task, Write about the following topic: Children who are brought up in families that do not have large amounts of money are better prepared to deal with the problems of adult life than children brought up by wealthy parents. To what extent do you agree or disagree with the opinion? Give reasons for your answer and include any relevant examples from your own knowledge and experience. Write at least 250 words.

Rating scales  Holistic (see handouts) + fast (practicality) – not easy to interpret results/ remember all to arrive at score  Analytical (see handouts) parameters (task fulfillment, content, organization, cohesion, register, accuracy, etc.) and criteria ( and their description) +clear, explicit, detailed diagnostic info, reliable, good for guiding learners -time consuming

Holistic rating scales Positive features: –practicality – fast; –rating holistically may be more naturalistic. Disadvantages: –no useful diagnostic information - single score; –not always easy to interpret - raters not required to use same criteria to arrive at score.

Analytic rating scales Positive features: can provide diagnostic information if scores reported separately; potentially clear, explicit and detailed; usually more reliable (multiple scores); useful in training raters to focus on our construct; potentially useful in guiding learners. Disadvantages: time-consuming; may overburden raters.

How to reduce subjectivity ? specification of the content of the assessment; adopting standard procedures; basing judgments on specific defined criteria; requiring multiple judgments; undertaking appropriate training; moderation of judgments; checking validity and reliability by analyzing assessment data.