Assessment Framework and Test Blueprint Margaret Wu

Development of a Framework and Test Blueprint Draft items Item Development Development of a Framework and Test Blueprint Draft items Item panelling (shredding!) Iterative process: Draft items to illustrate, and clarify and sharpen up framework. Framework to guide item development.

Framework and Test Blueprint-1 Clearly identify ‘Why’ you are assessing (Purpose) ‘Whom’ to assess (Population) ‘What’ to assess (Construct domain) What need to be reported Overall score Subscores? For individual students or for cohorts (by district, gender, school)? Define parameters for the test,e.g.: Duration of the test and test administration procedures Scoring/marking constraints; item formats Other issues: security, feedback.

Specifying the Purpose How will the results be used? Determine pass/fail, satisfactory/unsatisfactory Award prizes Provide diagnostic information Compare students Set standards Provide information to policy makers Who will use the information? Teachers, parents, students, managers, politicians

Specifying the Population Grade, age level. In an industry. Profession. Ethnicity/Culture/Language issues. Gender. Notion of population and sample. Sampling method: random, convenience Size of sample Validity of test results could depend on the population/sample you are assessing

Specifying the Construct Domain - Examples Familiarity with sport: What is meant by ‘sport’? Include “Gym”? “Taichi?” “Gymnastics”? In Australian contexts? Define Problem Solving: As viewed in the workforce Workforce competencies As viewed in the cognitive sciences Cognitive processes: decoding, reasoning, domain specific knowledge Interpersonal skills Negotiation/conflict resolution skills Leadership skills Work with people from diverse backgrounds

Specifying the Construct Domain – Examples Achievement domains: Content oriented: Number, measurement, data, algebra Competency oriented: Conceptual understanding, procedural knowledge, problem solving Taxonomy of educational objectives (Bloom’s taxonomy of learning outcomes): cognitive and affective.

Blooms’ Taxonomy - cognitive Knowledge Comprehension Application Analysis Synthesis Evaluation

Considerations in defining the Construct of a test Validity Consideration Does the construct cover what the test is claimed to be assessing? E.g., language proficiency: speaking, listening, reading, writing Measurement Consideration How well the specifications for a construct “hang together” to provide meaningful scores? The idea of “unidimensionality” On-balance judgement Boundaries are never clear

Test Blueprint Sufficiently detailed so that test developers can work from these specifications. Range of difficulty Target reliability Item format. Weights of sub-domains Test administration procedures Timing, equipment, resources Marking requirements

Test Blueprint – example (PISA Reading) Aspect % of test % constructed % MC Retrieving information 20 7 13 Broad understanding Interpretation 30 11 19 Reflecting on content 15 10 5 Reflecting on form Total 100

Report by individual student What will be reported Any subscore By content domains. If so, then there needs to be sufficient number of items for each content domain Report by individual student Need to work out measurement error: the accuracy with which each student is measured. Typically, we need a very long test to measure individuals accurately. This has an impact on the test length. Report by subgroups Ensure that sample size is sufficient to report by subgroups Report possible factors related to achievement Need to collect student background information

Define parameters for the test Duration of the test Has an impact on measurement error If the purpose of the test is for cohort reporting, the tests can be shorter. Total number of items needed How many items are needed to cover the content domain? As many as possible. The use of rotated test booklet design Allow sufficient number of items to cover the content domain Prevent cheating Moderate fatigue effect How many open-ended/constructed items The need to hire markers The cost of marking

Uses of Frameworks & Blueprints To guide item development Don’t ignore specifications. Cross-check with specs constantly. To ensure that there is a clear and well-defined construct that can be stable from one testing occasion to another. Different item writing team Parallel tests To establish test validity Design test -> Administer test -> Process test results -> Interpret test results The last stage of interpreting test results relies on the test specifications (test blueprints) to establish what is being tested.

Adapting/borrowing test blueprints from other assessment projects Assessment framework may not suit – different purposes of assessment. e.g., differences between PISA and TIMSS (curriculum based or competency based) Identify learning goals, and not just what is taught Look beyond the textbook May avoid rote learning Avoid the possibility of students memorising test items Talk to stakeholders Policy makers Employers Educators

Using rotated booklet design to cover content areas and counteract fatigue effect Block 1 Block 2 Block 3 1 C1 C2 C4 2 C3 C5 3 C6 4 C7 5 6 7

An example of rotated test booklet design Block 1 Block 2 Block 3 1 C1 C2 C4 2 C3 C5 3 C6 4 C7 5 6 7