Assessing to Support New Learning Outcomes: Assessing for Graduate Capability Acquisition Evaluations & Assessment 2003 Sally Kift, Assistant Dean QUT Law Faculty

 QUT Overview Acknowledge input of large team on Large Grant. What’s the assessment issue? What are the areas on which we are focussing and why? Developing an assessment framework Later session on this today Look briefly at the areas of concern. What we have achieved/still to do. Discussion.

 QUT What’s it all about? In embedding graduate capabilities, questions arose re Validity and reliability of traditional assessment practices and their transferability to the new imperatives of assessing authentic learning tasks. How can we assure the quality of our assessment of students’ capability development? The (tentative) answer is that - we should be able to assure the quality of assessment if it satisfies certain criteria (as distilled from the educational literature)

 QUT Hidden Large Grant to examine & address challenges of re- formulating assessment and feedback practices to assure their validity and reliability in this new teaching and learning environment. 4 project areas either problematic in their own right (because assessment has been hampered by perceptions of subjectivity or cultural bias) or problematic because of difficulties experienced in formulating valid and reliable assessment tasks (particularly in large group teaching and in flexible delivery modes).

 QUT In four project areas: - Identified 4 areas of capability development that have proved extremely challenging to assess – 1. Embedding Indigenous content and perspectives ; 2. Development of oral communication (esp negotiation, advocacy, tutorial participation & interviewing for internal and external cohorts); 3. Infusion of ethical values and knowledge ; 4. Teamwork in large classes (internal and external cohorts)

 QUT What does that mean? Developed assessment framework to provide theoretical basis for identifying models of best practice Found assessment practices in the project areas both w/i and o/s QUT. THEN – evaluate these assessment tasks against draft framework to distill relative strengths and weaknesses (some egs of evaluation on website) This with a view to developing best practice models for trialling in 2 nd stage Re Indigenous area, content reviewed as well Session later today

 QUT Eg: Oral communication 5 specific communication skills being addressed: Interviewing Tutorial performance Negotiation Advocacy OK Oral presentations New criteria sheet; and Instructions to tutors & students (in core 1 st yr) Improved negotiation exercise in core 2 nd yr New assessment criteria (core 1 st yr); ‘Student Assessment Handbook” for tutors & instructions for students; Student video (o/s grant)

 QUT Eg: Teamwork Best practice models being developed for – In the classroom Outside the classroom (live interactive groups) In the virtual classroom (predominately electronic contact) Consider whether assessing process or product If process models could include: Formative or summative; optional or compulsory Self, peer or teacher assessment Allocating individual or group marks

 QUT Eg: Ethics – Very challenging Connections with Indigenous project area Legal ed currently utilises Bloom taxonomy to measure cognitive domain achievements (ie “professional” aspect of ethical knowledge) Audit revealed only one unit in Law (cf Justice) as having structured form of assessment that includes both cognitive and affective components.  Have had to devise a matrix of criteria and standards …2/

 QUT Ethics – Reloading the Matrix The matrix of criteria and standards – referenced to observable characteristics or qualities in assessment materials which are themselves coupled to accepted taxonomies of achievement Attempt to maximise the dependability and reliability of assessment. Have used Facione & Facione (1994) and Bloom et al (1964)

 QUT The Matrix Criteria mapped to increasing levels of achievement (Bloom 1 to 6) - ie leftmost criteria reflect “low-level” Bloom achievement (eg knowedge, comprehension), while rightmost criteria reflect highest level of Bloom achievement (synthesis, evaluation) Standards mapped to increasing levels of achievement from Facione & Facione 1 –4 & referenced to observable & objective descriptors reflecting differences across Facione– eg “locates ALL rules’”(L4); “locates MOST rules” (L3); “locates SOME rules” (L2) & “does not locate rules” (L1) CRITERIA  WarnerBros

 QUT Re Indigenous Project Area (  With ors across QUT) What is an Indigenous perspective (cf content): Access understanding through – the cultural construction of whiteness (eg explore own racial position) & consultation with Indigenous persons (latter to move beyond problematising and essentialising) Assessment: eg reflective assessment to chart attitudinal change informed by knowledge development and incorporating higher order analysis. Also Indigenous employment and staff development

 QUT Anything else? Grant website a major dissemination strategy Ethics approval for the research undertaken by Project team members Staff survey and student focus group Pleasing no of pubs: eg 4xE1, 4xC1,15xATN Including Casual staff (2 RAs and training) Links across Faculty and across University Positively received 2003 Law Discipline Review

 QUT Still TO DO: A LOT!!! Trial the new materials Survey their success with students and staff and analyse Refine assessment framework Produce some training videos to assist in both teaching oral communication skills and in modelling application of assessment criteria.

 QUT The Pretty Website –

Assessing to Support New Learning Outcomes: Assessing for Graduate Capability Acquisition Evaluations & Assessment 2003 Sally Kift, Assistant Dean QUT Law Faculty