Webinar Series Competency based assessment 2 Aug 2017: 07:00AM UTC/GMT Assoc. Prof. Claire Palermo Monash University Just to let you know: By participating in the webinar you acknowledge and agree that: The session may be recorded, including voice and text chat communications (a recording indicator is shown inside the webinar room when this is the case). We may release recordings freely to the public which become part of the public record. We may use session recordings for quality improvement, or as part of further research and publications. Webinar Hosts Professor Geoff Crisp, PVC Education, University of New South Wales g.crisp[at]unsw.edu.au Dr Mathew Hillier, Office for Learning & Teaching, Monash University mathew.hillier[at]monash.edu e-Assessment SIG

Competency based assessment Taking a programmatic approach to assessment to ensure graduate employability Assoc. Prof. Claire Palermo Deputy Director, Monash Centre for Scholarship in Health Education & Department of Nutrition, Dietetics and Food Faculty of Medicine, Nursing and Health Sciences A bit about me Fellowship – best practice CBA in medical education was programmatic

What comes to mind when you hear programmatic approach to assessment?

Competency Based Education – Graduate Employability Consumer needs Workplace system Graduate Capabilities Curriculum Assessment Competency based education framework provide the foundation from which to develop the workforce. Competencies – outcomes to be achieved by the profession Should be informed by population health needs and health system needs Competencies inform – curricula and assessment This framework applies for preparation for the workforce but also to support the qualified workforce Dietitics we were one of the first professional to develop national competency standards (in 1992). However, this approach has not necessarily been fully transferred to the preparation of and continuing education of dietitians to be outcomes focussed. Competency based education represents a fundamental shift in educational philosophy and perspective. A number or factors that have impeded full implementation. If we are to truly have an impact on health and nutrition we need to strengthen our approaches. Today in addition to exploring the new competency standards I hope to present to you accepted “truths” that resonate in the profession I aim to dispel these truths by presenting evidence to the contrary. By letting go of these accepted practises and embracing change we may truly be able to transform the profession of the future. Frenk J. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010

Dreyfus & Dreyfus Development Model …the routine and considered use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection that are applied to a practice problem with the aim of improving outcomes. Expert/ Master Proficient Competent Advanced Beginner The Dreyfus model of skills acquisition describes that development of competence. Competence is the routine and considered use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection that are applied to a practice problem with the aim of improving (in the case of dietetics) nutrition outcomes. In the development and maintenance of competence we must recognize that skills develop over time, practice, experience and training. We expect different levels of competence for different skills and our journey to the development of skills is different. The reality is that competency development is linear at all as suggested by this model but rather looks something like this… or this… However, it is a point on the spectrum of performance. We should be striving towards excellence – competence isn’t enough! What is competence? “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection” (Epstein and Hundert 2002) “is a point on the spectrum of improving performance” (Khan, 2012) What is performance? “a composite of (clinical) cognitive, psychomotor and affective abilities (attitudes) of the individual alongside their non-clinical skills like team-work, situational awareness… personality traits.” (Khan, 2012) Novice Time, Practice, Experience Dreyfus SE and Dreyfus HL. 1980 Carraccio CL et al. Acad Med 2008;83:761-7 Acknowledgements: Eric Holombe

What is a programmatic approach to assessment? Focus is on outcomes Assessment looks at performance and application of knowledge and skills that are needed for employment Every student is expected to achieve explicit standards which are criterion referenced Differences in trajectory of progress during training are expected (developmental) Formative feedback to help students improve performance is a priority Learner responsibility and accountability are stressed

Programmatic Assessment - Conceptually Van der Vleuten et al (2012)

Any questions?

Programmatic Assessment - Practically Focus is on outcomes Define outcomes e.g. Competency standards, graduate attributes Assess a broad range of capabilities Assessment to focus on outcome rather than process

Programmatic Assessment - Practically Assessment looks at performance and application of knowledge and skills that are needed for employment

Programmatic Assessment - Practically Every student is expected to achieve explicit standards which are criterion referenced

Programmatic Assessment - Practically Differences in trajectory of progress during training are expected (developmental)

Programmatic Assessment - Practically Formative feedback to help students improve performance is a priority

Programmatic Assessment - Practically Learner responsibility and accountability are stressed

Twelve Tips for programmatic assessment Develop a master plan for assessment Develop examination regulations that promote feedback orientation Adopt a robust system for collecting information Assure that every low-stakes assessment provides meaningful feedback for learning Provide mentoring to learners Ensure trustworthy decision-making Organise intermediate decision-making assessments Encourage and facilitate personalised remediation Monitor and evaluate the learning effect of the programme and adapt Use the assessment process information for curriculum evaluation Promote continuous interaction between the stakeholders Develop a strategy for implementation VAN DER VLEUTEN et. al. Twelve Tips for programmatic assessment. Med Teach. 2015, 37: 641–646.

What is required for: A health professional to manage a patient with chronic disease? A teacher to deliver a multiplication maths lesson to year 2 students? An engineer to design structural steel for a new home?

Provide effective care to a person with chronic disease Example of designing a program of assessment What does it take for a health care professional to: Provide effective care to a person with chronic disease

Example of designing a program of assessment Work-place assessors feedback Work-place assessors feedback Work-place assessors feedback Case study report on patient with diabetes Oral communication exam Learning experiences

Challenges in Implementation of Programmatic Approaches Stakeholders Assessment policy and procedures Unit/Course structures ▪ Resistance to integrating curriculum – Silos and resistance to new approach to thinking ▪ A desire to use A single-best instrument ▪ Institutional requirements of assessment – Accrediting body requirements – Formative vs. Summative assessment requirements ▪ Resource and curriculum renewal requirements (time and resources) – Less is more (do a few things well) ▪ Student resistance to change (buy in essential)

Programmatic approaches to assessment supports Summary Programmatic approaches to assessment supports A focus on outcomes/authentic assessment Holistic judgement on criteria linked to capability Longitudinal, multiple pieces of assessment evidence Feedback central Student directed/led

Assoc. Prof. Claire Palermo

Webinar Series Webinar Session feedback Recording available With thanks from your hosts Professor Geoff Crisp, PVC Education, University of New South Wales g.crisp[at]unsw.edu.au Dr Mathew Hillier, Office of Learning & Teaching Monash University mathew.hillier[at]monash.edu Recording available http://transformingassessment.com e-Assessment SIG