Dr Anie Attan 26 April 2017 Language Academy UTMJB ITEM WRITING WORKSHOP Dr Anie Attan 26 April 2017 Language Academy UTMJB

To begin -Test Specifications A blueprint/guide to writing test items; validating tests; comparing scores Contains information on Duration of test Locating texts Type & number of texts Difficulty & length of texts Number & type of items/tasks Allocation of marks Weighting of marks Proportion of LOTS/HOTS questions

Test Specifications cont’n Scoring criteria Purpose of test Skills & abilities to be tested Context of target learning unit Characteristics of test-takers Characteristics of tasks

SAMPLE TEST SPECS ULAB 1122 FINAL EXAM ULAB 2122 FINAL EXAM

Test Format Duration – 60 / 90 / 120 minutes Weighting – 30% / 40% / 50% / 70% / 100% Number of questions – 2 / 5 / 10 /25 / 45 Type of questions – T/F, MCQ, Short Answer Question, Chart Completion, Sentence Completion, Transformation, Summary Writing, Essay Writing, Report Writing, Pair Activity, Group Interaction, etc. Type of texts -- length difficulty Genre Readability Index

Test Design Concerns What is it that we wish to measure? How can it be accurately measured?

What Is It That We Wish To Measure? The underlying knowledge and ability that students need to apply to carry out the tasks / to answer the questions The specific skills that students need to demonstrate in fulfillment of particular tasks / answering questions

Defining the Reading Construct A measure of students’ ability e.g. linguistic, discourse & strategic to comprehend and convey meaning as depicted in various texts with emphasis on: Comprehension effectiveness Grammatical correctness

Defining the Listening Construct A measure of students’ ability e.g. linguistic, discourse, strategic & sociolinguistic to comprehend and convey meaning as depicted in various texts with emphasis on: - Comprehension effectiveness Grammatical correctness

Defining the Speaking Construct A measure of students’ ability e.g. linguistic, discourse, strategic & sociolinguistic to use language in various domains with emphasis on: Content correctness Comprehension effectiveness Contextual appropriateness Communicative effectiveness Grammatical correctness

Defining the Writing Construct A measure of candidates’ ability e.g. linguistic, discourse, strategic & sociolinguistic to use language in various domains with emphasis on: - Content correctness Comprehension effectiveness Contextual appropriateness Communicative effectiveness Grammatical correctness

SAMPLE OF CONSTRUCT

Defining the Reading Abilities Comprehension – deriving meaning of words, phrases, sentences, extracting specific info, identifying main ideas, understanding linear & non-linear texts, understanding relationships, recognising cohesive devices; Application – predicting outcomes, applying a concept to a new situation; Analysis – understanding language functions, distinguishing relevant from irrelevant , distinguishing fact from opinion, drawing inferences, identifying roles & relationships; Synthesis – relaying ideas & concepts, following development of an argument, summarising information; Evaluation – appraising information, making judgments, drawing conclusions, recognising writers’ stance.

Defining the Listening Abilities Knowledge – recalling information, recognising main idea and details Comprehension – deriving meaning, paraphrasing Application – predicting outcomes, applying a new concept to a new situation Analysis – understanding language functions, distinguishing relevant from irrelevant, fact from opinion, drawing inferences, identifying roles & relationships Synthesis – following development of an argument, summarising information Evaluation – appraising information, making judgments, drawing conclusions, recognising speaker’s intention

Defining the Speaking Abilities Ideas & content – presenting relevant ideas, providing adequate content, showing mature treatment of topic Interaction – initiating, interrupting, prompting, negotiating, clarifying, explaining, comparing, contrasting, giving opinion, making suggestion Coherence – developing and organising ideas, using appropriate linking devices Word choice – using a range of vocabulary appropriate for intended purpose and audience Conventions & grammar – using grammatically correct language, display correct pronunciation, stress, intonation

Defining the Writing Abilities Ideas & content – generating & developing ideas, comparing & contrasting ideas, establishing cause & effect relationship, explaining & illustrating ideas, responding critically to prompts Organisation & structure – writing effective introduction and conclusion, developing coherent and cohesive paragraphs, collating and assessing information Word choice – use a range of vocabulary appropriate for intended purpose & audience, use appropriate cohesive devices Convention & grammar – use grammatically correct language, correct mechanics of writing, using appropriate voice

DEFINING THE ABILITIES for ULAB 1122 for ULAB 2122 Refer to CLO in course outline

FAQs -How can we measure it? What type of inputs/stimuli should we choose for candidates to process? Where can we locate the sources for the inputs? What type of topics do we focus on? What type of topics should we avoid? At what level of difficulty should the tasks be pitched at? How do we design the items? What is the scope of the task?

Choice of inputs/stimuli Concern: There should be a match of testing objectives with the skills specification Strategies: Match inputs with ability expected Currency of topic Relevance of topic Interesting/motivating Challenging Not biased to a particular group

Source of Inputs Journal articles Research reports Newspaper editorials/articles Magazine articles Organisational reports Academic books Electronic texts

Focus of Inputs Challenge the thinking capability of candidates Raise awareness of issues Difficulty of propositional and organisational features within the required Readability Index Have potential for analysis & synthesis Have potential for evaluation

Difficulty of Task Matches test specifications: Matches requirement for candidates to recognise relevant information from input Matches requirement for candidates to analyse & synthesise required information from input Matches requirement for candidates to apply reasoning skills Matches requirement for candidates to apply skill of evaluation

Designing the Task Analyse the inputs Identify issue, theme, role, relationships Match with requirements of test specifications One that candidates can recognise, apply, analyse, synthesise and evaluate Choose item type that best matches candidates’ ability in demonstrating skill

Delivering the Input and Items Input: rhetorical style -informative, analytical, persuasive, argumentative, narrative style; structured & coherent text Items: cover a range of skills, clearly & accurately worded, no overlap or over-testing of ability

Thank you Anie Attan