OUNL’s assessment model January the 10th 2006 Colin Tattersall & Henry Hermans

Positioning this work in TENCompetence ­WP 6,Task 3. ­Develop a formal specification model and supporting tools that combines new assessment types and the ones included in the IMS QTI (providing input for standards development). The current IMS QTI specification concentrates on classical forms of assessment (multiple choice, short answer, etc.). For competence development also other assessment techniques should be included in the specification (e.g. 360 degrees feedback, portfolio assessment). During the first 18 months activities will build on initial work already done by OUNL and CITO (citogroup.nl) to specify a formal integrative model, and tools will be developed. This will provide valuable input for the further development of the IMS QTI specification.“ ­12 man months available in the first 18 months ­Proposed in December 2005: ­OUNL (2), Giunti (1), CERTH (2), UB (2), UvA (2), SU (3)

Assessment vs testing ­Assessment = all the systematic methods that can be used to gather information and evidence about student properties, based on a process, a product or the progress of a student, for the purposes of certification, placement or diagnoses in formative and summative contexts. ­This definition includes classical tests, examinations and questionnaires, as well as newer types of assessment, such as competence-based assessment, portfolio assessment and peer assessment. ­Where instruction and assessment are considered as separate activities, assessment is often referred to as ‘testing’ ­Taken from Joosten-Ten Brink et al (submitted)

Relevance for TENCompetence ­The judgment that a someone is competent in a domain can be achieved by using several assessment instruments ­Challenge: select the assessment types that yield the appropriate evidence

Some questions I have heard ­Why not just use QTI? ­Why not just use QTI+LD? ­What about other initiatives, eg FREMA? ­…

Why an(other) assessment model? ­Need for exchange of assessments ­development of reliable and valid assessments is time- consuming and expensive ­Discrepancy between: ­state of the art in assessment ­current standards (QTI)

Where and how is QTI lacking? ­tendency ­away from massive, standardized, summative testing with multiple choice questions based on knowledge acquisition. ­towards assessment integrated in learning and instruction, process-based, with student involvement. ­QTI focuses on those assessment types for which an unambiguous definition in technical terms can be specified. ­More needed: assessment process (which steps are to be carried out and by whom), but also rationale (what is being tested and how)

What about using QTI+LD ­Can do more, but much remains implicit, not described in the language of assessment

Goals of original OUNL work ­Develop conceptual model of assessment ­Cover state of the art in assessment ­Create foundation for a new standard ­Document and evaluate methodology ­UML as notation ­Model expert knowledge

Project information ­Duration: 1 year ­Partners: ­Open University of the Netherlands ­Citogroep Arnhem ­Assessment experts: ­Citogroep ­Open University ­Fontys

Project phases ­Initial model (expert sessions) ­Internal evaluation ­Several (>70) change requests ­Modified version ­Layered model ­Broad coverage of assessments

Towards a bird’s-eye view....

Phases in the assessment process Assessment Design Item Construction Assessment Construction Assessment Delivery Respons Evaluation Decision Making

Assessment design

Item construction

Assessment construction

Assessment run

Respons processing (1)

Respons processing (2) IndicatorScore

Decision making

To be discussed Is it already done then? ­Evaluation ­What about other initiatives (FREMA)? ­Next steps ­How might this integrate into the TENCompetence infrastructure? ­Tooling? ­Relationship with IMS ­TENCompetence explicitly mentions new QTI version (although 2.1 is underway)