1. Paul Kleiman PALATINE Lancaster University Chaos and Crystals: negotiating assessment or Assessing at the edge of chaos

2. Chaos and Complexity Performance Theory Personal Experience Research Paradigms Assessment

3. Chaos and Complexity Performance Theory Personal Experience Research Paradigms Assessment

4. SOME STARTING POINTS

5. Performer/Teacher Text/Curriculum Audience/ Students Performance/ Learning Space Assessment

6. c h a o s a n d c o m p l e x i t y

7. Self-organisation Paradox Emergence The edge of chaos

8. operating on the edge of chaos STASIS Close to certainty Close to agreement Far from agreement Far from certainty (Based on Stacey 2000) zone of complexity CHAOS Edge of Chaos – tension AND creativity Simple Complicated Complex

10. Students Failure Finance Staff Reality HEI Administration Resits Paperwork generator Learning & Teaching Understanding Research External Examiner Graduates Stress Knowledge Skills

11. Students Failure Finance Staff Reality HEI Administration Resits Paperwork generator Learning & Teaching Understanding Research External Examiner Graduates Stress Knowledge Skills Closed system Knowledge as external and transmitted Predictability Linear causality Deterministic

13. the problem The more assessment involves qualitative information, the more subjectivity is involved Stricter assessment criteria + more structured and proscribed content = improved reliability BUT….. above would “obliterate the essence of qualitative assessment in terms of flexibility, personal orientation and authenticity”. Driessen et al (2005) Medical Education 39: 214–220

14. Positivist Aim Explanation, control Knowledge Objective, measurable, value-free, universal, generalisable, external, quantifiable, can be transmitted and acquired Teacher/Researcher Expert, independent Criteria Rigour via ‘holy trinity’ of validity, reliability and generalisability There’s a reality ‘out there’ that can be studied, captured and understood

15. Interpretive Aim Understanding Knowledge Subjective, contextualised, value-dependent, multiple-realities Teacher/Researcher Independent participant Criteria ‘Rigour’ (?) via credibility, transferability, dependability, confirmability ‘Truth’ is a matter of consensus amongst informed and sophisticated constructors, not of correspondence with an objective reality.

16. Interpretive ‘Facts’ have no meaning except within some value framework; hence there cannot be an ‘objective’ assessment of any proposition. Phenomena can be understood only within the context in which they are studied; findings from one context cannot be be generalised to another; neither problems nor their solutions can be generalised from one setting to another. Note: the problem with ‘evidence-based practice’, ‘best practice’, ‘transferable skills’, ‘benchmarks’, ‘national standards’, etc.

17. Therefore: Find and replace Replace Validity with Credibility, Coherence, Consistency, Trustworthiness, Authenticity Certainty with Relativity Generalised Explanation with Local Understanding Source Data with Empirical Materials (which can become data…. and then evidence) Is it true? with Does it work? Single Point Perspective with Multiple Perspective The Triangle with the Crystal

18. The complex and multidimensional nature of learning cannot be captured effectively and comprehensively by any single instrument or analytical procedure. The complex and multidimensional nature of learning cannot be captured effectively and comprehensively by any single instrument or analytical procedure. The use of multiple methods reflects an attempt to secure an in-depth understanding of the phenomenon in question. However…..

19. from TRIANGULATION to CRYSTALLISATION

20. Triangulation carries with it the image of a mathematical procedure adding rigour and discipline in one sense, but in another sense restricting the research to one of scientific method and a positivist framework where variables are few and can be controlled or manipulated. (Chien, 2004) A B C P

21. Ad sedere / to sit down together "You've got to involve students actively, not just view them as objects of assessment, but as agents of assessment. This can be done in many ways. One is that you ask students systematically what they have learned. It's a simple idea; it's rarely done.....You find that students say some remarkable things.” Walt Haney, Professor of Education, Center for the Study of Testing, Evaluation and Educational Policy, Boston, USA. negotiating assessment

22. Presentation and Performance: Negotiated assessment Students engaged in creative practice work at different levels AND different ways. The products they create will be different, as will the processes and methods utilised, and the disciplines they represent That assessment should operate and be perceived as an integral part of the learning process rather than 'bolted- on' to the end of that process. That the form, content and implementation of the assessment process should be commensurable with the discourse and practices of the field negotiating assessment

23. Six assessment fields: Presentation/Production i.e. the finished product presented to an audience Process i.e. the journey that led to the product Idea i.e. the ideas that informed both the process and the product. Technical i.e. the quality and utility of the technical features of the product and the skills with which they were assembled and/or operated Documentation i.e. research, design, sketches, planning, evaluation, analyses, portfolio, etc. Interview i.e. the student's ability to articulate their understanding, utilisation and application and use of any of the above. negotiating assessment

24. Learning contract – negotiated Regular meetings/tutorials Assessors see the performance/presentation + student compiles ‘portfolio of evidence’ Assessment tutorial c. 30 min; individual / group + at least 2 assessors Work through the criteria - moving upwards Reach a point of maximum information, optimal achievement Agree a grade band Sense of ‘completeness’, ‘accomplishment A learning experience for all concerned negotiating assessment

25. ? p.kleiman@lancaster.ac.uk