1. Overview: Competency-Based Education & Evaluation
Dr. Deborah M. Fournier
Assistant Provost for Institutional Research & Evaluation
Boston University Medical
December 7, 2009

2. Competency-based Education
An approach to curriculum that emphasizes the attainment of specific knowledge, skill and attitudes associated with carefully defined performance behaviors expected of entry-level independent practitioners.

3. Competency-based Education
Performance evolves along a continuum from novice  beginner  competent  proficient  mastery

4. Competency-based Education
Focus on integration and synthesis of information and experiences to create “working knowledge” that can be used in a real world setting

5. Competency-based Education
Focus on how students organize and structure knowledge (how students learn versus how and what to teach)

6. Competency-Based Education
Grounded in Mastery Learning
A Mastery Learning approach can produce significant gains in student achievement

7. Research One-on-One Coaching and Apprenticeship:
90% of students perform at the level primarily reached by the top 16% of students in a traditional curriculum
Mastery Learning Strategies:
Given enough time on task and under optimal conditions 70% of students achieve the level primarily reached by the top 16% of students in a traditional program
Longer lasting effect weeks after the course ended (recall was enduring)

8. Aptitude

9. Achievement

10. Mastery Learning

11. Major Assumptions
The higher the quality of instruction, the less relevant student aptitude becomes
Most students can learn well and achieve high performance if:
1) given enough time on task
2) instruction is designed and delivered for “optimal effect”

12. Major Assumptions Traditional Time = Constant Learning = Variable
Competency-based
Time = Variable
Learning = Constant

13. General Characteristics
Competency-based
Cognitive
Outcomes-based
Demonstrating independent practice
Successive stages
toward mastery
Each student gets the amount and kind of instruction needed
Instruction varies and learning is constant
Achievement is skewed
to the right
Traditional
Behavioral
Content-based
Meeting clinical
requirements
Accumulation of facts, concepts, skill
All students receive
same identical
instruction
Instruction is constant and learning varies
Achievement is normally distributed

14. Basic Features Competencies Prerequisite Testing
Instructional Activities
Formative Evaluation
Summative Evaluation
Criterion-based Evaluation
Remediation or Corrective Activities
Enrichment Activities
Milestones (Progress Monitoring Mechanisms)

15. Competency-based Evaluation
Involves the evaluation of complex, higher-level skills in a real world setting that provides open-ended tasks that require a high degree of choice and judgment.

16. Competency-based Evaluation
Multiple methods
Multiple measures
Multiple perspectives
Measurement over time
Pattern of evidence
Levels of evidence
Valid and reliable measurement

17. Formative Evaluation
To direct and pace student learning and tailor faculty teaching
It gives evidence on how students are changing in their progress toward competency
It gives evidence about learning while instruction is in progress
Encourages students to take intellectual risks without being penalized

18. Summative Evaluation
It gives evidence on how students have changed (as opposed to formative which shows how they are changing)
The student is put in a testing mode where they are performing independently, that is, there is “no instruction/teaching in progress” as with formative

19. Evaluation Tools Standardized Patients
Objective Structured Clinical Examination
Simulations
Oral Structured Examination
Triple Jump Examination
Chart-stimulated Recall
Skill Test
Competency Examination
Portfolio Review
Case Presentation
Mock Board
Written Tests (combined w/ performance tests)

20. Standardized Patients
Individuals (actors) who are scripted to behave in a certain way so as to solicit certain kinds of responses from students
Uses a checklist type instrument called an Interaction Analysis tool where a faculty observer records and scores how the student behaves/performs in a particular scenario
Useful for assessing communication and interpersonal skills, diagnosis, treatment planning, and patient presentations

21. Objective Structured Clinical Exam
A series of timed stations that examine a variety of higher-order thinking situations
Write a tx plan, assessment and diagnosis, interview a patient
Useful for assessing diagnosis, treatment planning, clinical reasoning

22. Simulations
Contains some of the most salient features of the situation to be experienced by students once in practice
Computer sims, manikins, actors, models
Useful for assessing subsets of clinical skills, clinical judgment

23. Oral Structured Exam
A series of standardized questions that require the student to use higher-order thinking skills to respond
Questions focus on patient management, treatment planning, cultural awareness, outcomes of treatment
Useful for assessing judgment, professionalism, evidenced-based critique

24. Triple Jump Exam
A specific type of Oral Structured Exam that involves three steps:
1) Problem Definition (1/2 hour)
2) Information Search (2 hours)
3) Synthesis (1/2 hour)
Useful for assessing critical thinking, problem solving, research skills, ethical decision-making, self-directed learning

25. Chart-stimulated Recall
A review of several patient charts that belong to a student
Faculty evaluator asks a series of standardized questions on patient mgt, risk assessment, clinical judgment
Useful for assessing patient management, ethical decision-making, cultural awareness

26. Skill Test
Examines an extracted set of skills from a more complex situation and ignores other skills
Emphasis is often on evaluating the product at the end of each step
Useful for assessing a variety of skill sets in the novice and beginner stages

27. Competency Exam
Integration of multiple skill sets while the student is rendering patient care
Focused on observing and scoring the integration of a broad range of skills during the entire student-patient interaction
Uses open-ended tasks that require a high degree of choice and judgment
Useful for assessing integrated knowledge and skill – putting it all together in varied and complex situations

28. Portfolios
A “purposeful” collection of student work that documents evidence of progress and achievement over time
Not a collection of most assignments completed during school
Requires careful development of guidelines for what to include, standards, timelines, etc.
Useful for assessing many skills evidenced in other tools