1. Dr Anie Attan 26 April 2017 Language Academy UTMJB
ITEM WRITING WORKSHOP
Dr Anie Attan
26 April 2017
Language Academy UTMJB

2. To begin -Test Specifications
A blueprint/guide to writing test items; validating tests; comparing scores
Contains information on
Duration of test
Locating texts
Type & number of texts
Difficulty & length of texts
Number & type of items/tasks
Allocation of marks
Weighting of marks
Proportion of LOTS/HOTS questions

3. Test Specifications cont’n
Scoring criteria
Purpose of test
Skills & abilities to be tested
Context of target learning unit
Characteristics of test-takers
Characteristics of tasks

4. SAMPLE TEST SPECS
ULAB 1122 FINAL EXAM
ULAB 2122 FINAL EXAM

5. Test Format Duration – 60 / 90 / 120 minutes
Weighting – 30% / 40% / 50% / 70% / 100%
Number of questions – 2 / 5 / 10 /25 / 45
Type of questions – T/F, MCQ, Short Answer Question, Chart Completion, Sentence Completion, Transformation, Summary Writing, Essay Writing, Report Writing, Pair Activity, Group Interaction, etc.
Type of texts
-- length
difficulty
Genre
Readability Index

6. Test Design Concerns What is it that we wish to measure?
How can it be accurately measured?

7. What Is It That We Wish To Measure?
The underlying knowledge and ability that students need to apply to carry out the tasks / to answer the questions
The specific skills that students need to demonstrate in fulfillment of particular tasks / answering questions

8. Defining the Reading Construct
A measure of students’ ability
e.g. linguistic, discourse & strategic to comprehend and convey meaning as depicted in various texts with emphasis on:
Comprehension effectiveness
Grammatical correctness

9. Defining the Listening Construct
A measure of students’ ability
e.g. linguistic, discourse, strategic & sociolinguistic to comprehend and convey meaning as depicted in various texts with emphasis on:
- Comprehension effectiveness
Grammatical correctness

10. Defining the Speaking Construct
A measure of students’ ability
e.g. linguistic, discourse, strategic & sociolinguistic to use language in various domains with emphasis on:
Content correctness
Comprehension effectiveness
Contextual appropriateness
Communicative effectiveness
Grammatical correctness

11. Defining the Writing Construct
A measure of candidates’ ability
e.g. linguistic, discourse, strategic & sociolinguistic to use language in various domains with emphasis on:
- Content correctness
Comprehension effectiveness
Contextual appropriateness
Communicative effectiveness
Grammatical correctness

12. SAMPLE OF CONSTRUCT

13. Defining the Reading Abilities
Comprehension – deriving meaning of words, phrases, sentences, extracting specific info, identifying main ideas, understanding linear & non-linear texts, understanding relationships, recognising cohesive devices;
Application – predicting outcomes, applying a concept to a new situation;
Analysis – understanding language functions, distinguishing relevant from irrelevant , distinguishing fact from opinion, drawing inferences, identifying roles & relationships;
Synthesis – relaying ideas & concepts, following development of an argument, summarising information;
Evaluation – appraising information, making judgments, drawing conclusions, recognising writers’ stance.

14. Defining the Listening Abilities
Knowledge – recalling information, recognising main idea and details
Comprehension – deriving meaning, paraphrasing
Application – predicting outcomes, applying a new concept to a new situation
Analysis – understanding language functions, distinguishing relevant from irrelevant, fact from opinion, drawing inferences, identifying roles & relationships
Synthesis – following development of an argument, summarising information
Evaluation – appraising information, making judgments, drawing conclusions, recognising speaker’s intention

15. Defining the Speaking Abilities
Ideas & content – presenting relevant ideas, providing adequate content, showing mature treatment of topic
Interaction – initiating, interrupting, prompting, negotiating, clarifying, explaining, comparing, contrasting, giving opinion, making suggestion
Coherence – developing and organising ideas, using appropriate linking devices
Word choice – using a range of vocabulary appropriate for intended purpose and audience
Conventions & grammar – using grammatically correct language, display correct pronunciation, stress, intonation

16. Defining the Writing Abilities
Ideas & content – generating & developing ideas, comparing & contrasting ideas, establishing cause & effect relationship, explaining & illustrating ideas, responding critically to prompts
Organisation & structure – writing effective introduction and conclusion, developing coherent and cohesive paragraphs, collating and assessing information
Word choice – use a range of vocabulary appropriate for intended purpose & audience, use appropriate cohesive devices
Convention & grammar – use grammatically correct language, correct mechanics of writing, using appropriate voice

17. DEFINING THE ABILITIES
for ULAB 1122
for ULAB 2122
Refer to CLO in course outline

18. FAQs -How can we measure it?
What type of inputs/stimuli should we choose for candidates to process?
Where can we locate the sources for the inputs?
What type of topics do we focus on?
What type of topics should we avoid?
At what level of difficulty should the tasks be pitched at?
How do we design the items?
What is the scope of the task?

19. Choice of inputs/stimuli
Concern:
There should be a match of testing objectives with the skills specification
Strategies:
Match inputs with ability expected
Currency of topic
Relevance of topic
Interesting/motivating
Challenging
Not biased to a particular group

20. Source of Inputs Journal articles Research reports
Newspaper editorials/articles
Magazine articles
Organisational reports
Academic books
Electronic texts

21. Focus of Inputs Challenge the thinking capability of candidates
Raise awareness of issues
Difficulty of propositional and organisational features within the required Readability Index
Have potential for analysis & synthesis
Have potential for evaluation

22. Difficulty of Task Matches test specifications:
Matches requirement for candidates to recognise relevant information from input
Matches requirement for candidates to analyse & synthesise required information from input
Matches requirement for candidates to apply reasoning skills
Matches requirement for candidates to apply skill of evaluation

23. Designing the Task Analyse the inputs
Identify issue, theme, role, relationships
Match with requirements of test specifications
One that candidates can recognise, apply, analyse, synthesise and evaluate
Choose item type that best matches candidates’ ability in demonstrating skill

24. Delivering the Input and Items
Input: rhetorical style -informative, analytical, persuasive, argumentative, narrative style; structured & coherent text
Items: cover a range of skills, clearly & accurately worded, no overlap or over-testing of ability

25. Thank you Anie Attan