Orvill Adams, Orvill Adams & Associates B.V. Orvill Adams Orvill Adams & Associates B.V. Measuring the Products of Medical Education

Orvill Adams, Orvill Adams & Associates B.V. Core Competencies for Canadian Physicians CanMEDS Medical Expert Communicator Collaborator Manager Health Advocate Scholar Professional 2

Orvill Adams, Orvill Adams & Associates B.V. Success Indicators Measure integration of CanMEDS into current curriculum (gap analysis) Conduct an inventory of current databases Determine critical partners Design and implement the data integration 3

Orvill Adams, Orvill Adams & Associates B.V. Success Indicators Measure why & how admissions filters (+/ ‐ pre ‐ admission programs) impact outcomes with respect to: discipline choices; location; quality; retention; etc... Develop and implement tools to measure objective ‐ driven standardized outcomes/competencies across a variety of learning settings/models Determine impact of financial accessibility issues on applicant pool and choices of matriculates. Establish mechanism for engaging all stakeholders in informing curricular decisions within each faculty of medicine. Determine how sites and teachers are currently selected prepared and evaluated. Assess characteristics of applicants and non ‐ applicants that predict career choice 4

Orvill Adams, Orvill Adams & Associates B.V. Questions and Approach Based on results from 41 United States & Canadian Medical Schools How do you measure the contribution of faculty? Developing a system of measurement Quantifying the educational activities 5

Orvill Adams, Orvill Adams & Associates B.V. Sources of information /data Literature review Informal discussions and networking An questionnaire sent to 125 US and 16 Canadian Medical Schools 41 Schools or Departments had developed measurement instruments 6

Orvill Adams, Orvill Adams & Associates B.V. Data Collected Type of educational metric system being used Length of time it had been in place Process used to develop the method Types of information captured Intended outcomes of the system Reactions to the system of faculty members, chairs and school administrators Lessons learned by primary informants 7

Orvill Adams, Orvill Adams & Associates B.V. Reason for developing a measuring system To develop a systematic and rational approach to distribute resources Track the amounts of resources spent on teaching and other education related activities To dispel mistrust – e.g. pools of funds, favorites To counteract the myth that faculty cannot to teach or are not compensated to do so To make the educational mission more visible 8

Orvill Adams, Orvill Adams & Associates B.V. Tracking systems A contract hour method- Actual contact time - Actual contact time plus time for preparation and time for evaluation A relative value method- Different teaching activities have different resource use Clinical teaching - how much time does the teaching add to the day of the clinical teacher? Is productivity lost by the clinical teacher because of presence of students? 9

Orvill Adams, Orvill Adams & Associates B.V. Understanding why the data is collected Leaders – Present the institutional story about change and improvement Faculty, managers – What problems can the data be used to solve? How will these data be used to improve the University? What will be different about the medical school in the future? How will the data be used? 10

Orvill Adams, Orvill Adams & Associates B.V. Lessons learned – cautions Faculty need to be included and informed Fear of leadership by numbers In – search of perfect measure Faculty must understand the use of data Use simple understandable measurement approach W. Mallon – Academic Medicine,

Orvill Adams, Orvill Adams & Associates B.V. An new approach to measuring research The Research Excellence Framework (UK) provide authoritative and comprehensible ratings of research excellence in all disciplines in HEIs across the UK inform the UK funding bodies’ allocation of grant for research, as determined by each of the four funding bodies (each year the UK funding bodies allocate around £1.76 billion for research) provide useful information and benchmarks about research excellence – both for the public and for institutions provide accountability for public expenditure on research in higher education. 12

Orvill Adams, Orvill Adams & Associates B.V. The research excellence framework contd. “The REF will assess research excellence through a process of expert review. It will be based on HEIs submitting evidence of their research activity and outcomes, to be assessed by expert panels” 13

Orvill Adams, Orvill Adams & Associates B.V. Areas of assessment Outputs – The primary focus of the REF will be to identify excellent research of all kinds, as demonstrated by the quality of research publications and other outputs. Impact – Significant additional recognition will be given where researchers have built on excellent research to deliver demonstrable benefits to the economy, society, public policy, culture or quality of life. Environment – REF will also assess how far the research environment supports a continuing flow of excellent research and its effective dissemination and application 14

Orvill Adams, Orvill Adams & Associates B.V. Units of Assessment 1 Clinical Medicine Public Health, Health Services and Primary Care Allied Health Professions, Dentistry, Nursing and Pharmacy Psychology, Psychiatry and Neuroscience Biological Sciences Agriculture, Veterinary and Food Science Earth Systems and Environmental Sciences Chemistry Physics Mathematical Sciences Computer Science and Informatics Aeronautical, Mechanical, Chemical and Manufacturing Engineering Electrical and Electronic Engineering, Metallurgy and Materials 15

Orvill Adams, Orvill Adams & Associates B.V. Units of assessment 2 Civil and Construction Engineering General Engineering Architecture, Built Environment and Planning Geography, Environmental Studies and Archaeology Economics and Econometrics Business and Management Studies Law Politics and International Studies Social Work and Social Policy Sociology Anthropology and Development Studies 16

Orvill Adams, Orvill Adams & Associates B.V. Units of assessment 3 Education Sport and Exercise Sciences, Leisure and Tourism Area Studies Modern Languages and Linguistics English Language and Literature History Classics Philosophy Theology and Religious Studies Art and Design: History, Practice and Theory Music, Drama, Dance and Performing Arts Communication, Cultural and Media Studies, Library and Information Management 17