1 HOW TO BE A GOOD COURSE DIRECTOR PART 2 - HOW TO IMPROVE A COURSE Master Teacher Guild Mike Fennewald, PhD (SGPDS) Daniel Bareither, PhD (SCPM) Lecia Apantaku, MD (CMS) Roberta Henderson, PT, PhD (CHP) Fred Sierles, MD (CMS) John Becker (SCPM) December 15, 2011

2 WHO IS RESPONSIBLE FOR IMPROVEMENT? Course Director: – This individual has been selected by the department and the college to coordinate the learning activities for the course (or clerkship). They must understand that the course is not THEIRS but rather they have the responsibility for providing an excellent educational experience for the students. They are responsible for the selection and evaluation of the faculty that participate in the course. Department Chairs: – Responsible for selecting qualified faculty to serve as course directors and for reviewing their performance.

3 WHO IS RESPONSIBLE FOR IMPROVEMENT? Deans: – Responsible for making sure the departments are providing excellent educational experiences, that there is assessment of the curriculum, and that the students are achieving the learning outcomes. Assessment Committee: – Responsible for coordinating the methods of assessment used to collect data on the curriculum, the analysis of the data, and identifying any problems and making recommendations to the Curriculum Committee and Dean. Curriculum Committee: – Responsible for management of the curriculum and taking action regarding recommendations from the Dean and Assessment Committee.

4 WHAT INFORMATION CAN BE USED FOR IMPROVEMENT? Course/Clerkship Evaluations - student input regarding: – Organization of the course/clerkship – Learning objectives – The learning outcomes that the course/clerkship helps to develop – Method of evaluation – Use of learning management system (D2L) – Integration of basic science and clinical science information – Faculty availability – Faculty contribution to their understanding of the course material – Constructive comments Note: Process for review and feedback from evaluations (assessment committee).

5 WHAT INFORMATION CAN BE USED FOR IMPROVEMENT? Process: – Development of the course evaluation (assessment and curriculum committees) – Determine method for students to complete course evaluation – Evaluation results compiled by someone other than the course director – Evaluation results provided to the department chair – Department chair reviews evaluation results with the course director and recommends any actions to be taken – Evaluation results and recommended actions forwarded to the Assessment Committee and Dean – Assessment Committee makes recommendations to the Curriculum Committee and/or Dean – If recommendation is to the Curriculum Committee then they make a recommendation to the Dean. Note: Correspondence from Dr. Marinelli regarding review of the graduate curriculum and the Oversight Committee

6 WHAT INFORMATION CAN BE USED FOR IMPROVEMENT? Peer Evaluations – faculty input regarding: – Course materials available to students – Established learning objectives – Quality of visual materials – Instructor – knowledge, interaction with students, utilization of AV, professional conduct – Presentation – organized, understandable, emphasized major points, integrated basic science and clinical information, delivered at a good pace, timely Note: Need a process for review and feedback from evaluations (department chair).

7 WHAT INFORMATION CAN BE USED FOR IMPROVEMENT? Assessment Committee – identifies a variety of internal and external methods to assess the achievement of the student learning outcomes/competencies, and the achievement of the programmatic outcomes, and curriculum/teaching: – Internal assessment - ongoing performance in a course (individual exams) overall course performance (grade distribution) compared to other courses occurring at the same time mean of the overall course performance compared to previous years performance on comprehensive exams course evaluations curriculum evaluations peer evaluations – External assessment - performance on board exams performance in clerkships performance in residency performance on licensure exams

8 WHAT INFORMATION CAN BE USED FOR IMPROVEMENT? Curriculum Committee: – major mechanism by which the faculty develop and review the curriculum – coordinates the process for course evaluations – coordinates the review of the syllabi, the methods of instruction, the scheduling of courses, the size and sequencing of courses, and receives and evaluates course proposals – annual evaluation of the curriculum to look at the sequencing of courses, the integration of basic science and clinical science information, the development of critical thinking, and the utilization of the learning management system – develops and reviews academic policies Student Focus Groups: – focus groups of students can be used to obtain additional information on a particular issue identified in the assessment results

9 WHAT INFORMATION CAN BE USED FOR IMPROVEMENT? Curriculum Committee: – major mechanism by which the faculty develop and review the curriculum – coordinates the process for course evaluations – coordinates the review of the syllabi, the methods of instruction, the scheduling of courses, the size and sequencing of courses, and receives and evaluates course proposals – annual evaluation of the curriculum to look at the sequencing of courses, the integration of basic science and clinical science information, the development of critical thinking, and the utilization of the learning management system – develops and reviews academic policies

10 FACULTY PERFORMANCE AND PROBLEMS Prevent problems. Monitor teaching closely. Continue to orient, mentor and monitor. Expect, praise and reward improvement. If a few cycles fail to yield improvement, reassign the faculty member.

11 Prevent Problems (Primary Prevention) Primary prevention. Prevent problems before they occur. – Consider teaching ability when hiring. – Assign classes to best teachers Not necessarily most expert on the topic – Orient and mentor new teachers. Prepare new teachers for first class.

12 Primary Prevention (cont.): Reward Good Teaching Reward Good Teaching – Offer praise, encouragement. – Suggest—to your chair—raises, travel support, promotion, tenure. – Write recommendation letters. – Nominate for teaching awards. – In considering tenure, your chair (with your input) should ask, “What will he or she do if he or she loses this grant?”

13 Monitor Teaching Closely For new or “struggling” teachers – Attend, help set up, give “tech support” for First few lectures, some (or all) subsequent lectures Early small group, occasional subsequent small groups

14 Monitor Teaching Closely: Prompt Oral Feedback Give prompt oral feedback. – During a break – Soon after class (same day, next couple of days) How to give feedback. – Begin with praise. – Empathize with faculty member’s feelings, perceptions and experiences in the class. – Follow with constructive criticism. – Summarize hopefully, end on a positive note.

15 Monitor Teaching Closely: Review Written Feedback Obtain, read, distribute and review with faculty member – Student evaluations of teaching, test item preparation – Student exam performance on faculty member’s topic Department exams National exams

16 Monitor Teaching Closely: Keep Chair Posted; Maintain Records Keep chair apprised of, and give chair access to, above indicators. Obtain and maintain records of feedback received by faculty member. – Documentation crucial if faculty member challenges decisions based on his or her teaching.

17 Follow-up Beyond Early Interactions and Feedback Continue to orient, mentor, monitor. Expect and praise each effort and success at improvement. If sub-par teaching and failure to improve continue after two or three cycles of monitoring and feedback, – Suggest, to chair, reassignment of faculty member to other duties.

18 Course Director Vignettes What Would You Do in Each Case?

19 Vignette 1 The Course Director is attending a two hour lecture presented by an early-career faculty member who is bright, confident and flexible. During the first 25 minutes of the first 50 min., the lecturer gives multiple colorful, vivid, relevant illustrations of patients she has treated, regularly poses questions to the class, and the class is attentive and energized. Inexplicably, during the second 25 min. of the first 50 min., the lecturer provides no illustrations, poses no questions, and many class members—as well as the Course Director, nod off periodically to sleep. The Course Director should…

20 Vignette 2 A part-time associate professor of psychiatry with nine years’ teaching experience gives a two hour, first-time, one- presentation per year lecture on a topic on which she is expert. She presents the topic at the intellectual level of an introductory college psychology class, a topic that is already “known by heart” by most of the students. Her voice is monotonous. She poses no questions to the class, takes no questions, and gives only two illustrative examples of patients she has seen. During the class, several students— and the Course Director—“nod off.” Post-course, the students give her low ratings, and their narrative comments point out the weak points noted above. The Course Director should…

21 Vignette 3 In the anonymous student feedback following a third year required psychiatry clerkship, two students give an assistant psychiatry professor an overall rating of “F,” and each of them mentions— in the narrative—something like, ”Just awful. The only thing he had us do was draw blood for his lab experiments. He gave us no patient care responsibilities and spent virtually no time teaching us.” This fits with this faculty member’s previous behavior. The Clerkship Director should…

22 Vignette 4 In the written anonymous student feedback following a third year required psychiatry clerkship, five of the 23 clerks spontaneously mention, in their narrative comments, something like: “Resident Dr. A gave us awesome off-the- cuff mini-lectures, and is a terrific supervisor and clinician.” Prior to his psychiatry residency, Dr. A published 17 psychiatry papers in refereed journals. The Clerkship Director should…

23 STUDENT PERFORMANCE Using internal measures to adjust curriculum Exams Clinical Performance End of course evaluations Using external measures to adjust curriculum National Board of Medical Examiners (NBME) subject exams Residency director questionnaires Medical School Graduation Questionnaires

24 NBME SUBJECT EXAMINATION CONTENT AREA ITEM ANALYSIS REPORT

25 NBME STEP I

26 Medical School Graduation Questionnaire 2011 All Schools Summary Report

27 MEDICAL SCHOOL GRADUATION QUESTIONNAIRE

28 MEDICAL SCHOOL GRADUATION QUESTIONNAIRE

29 M1 AND M2 QUESTIONNAIRES Please rate the professionalism with which you were treated in this course by the course faculty. Excellent, Good, Adequate, Unacceptable, Cannot evaluate Please rate the professionalism with which other students were treated in this course by the course faculty. Please rate the professionalism with which you were treated by your fellow students in this course. Additional comments on professionalism in this course: Lecturers and Notes/Labs: Please rate Dr. _______’s lectures: Please rate Dr. _______’s notes and related materials (e.g. audio, video, PowerPoint): Constructive comments.

30 M1 AND M2 QUESTIONNAIRES I attended about _______ of the lectures: The D2L site for this course was: Excellent Good Adequate Unacceptable Cannot evaluate The clinical relevance of this course was: The effectiveness of the exams to assess course objectives was: Integration of course content with other courses was: This course overall was:

31 M3 QUESTIONNAIRE Question 1 Please rate how well the specific clerkship objectives and clerkship policies (e.g. grading criteria, examination format) were identified early in the clerkship. Superior Very Good Acceptable Deficient Question 2 Please rate how well significant facts, principles, and basic science concepts in this clinical area were explained. Question 3 Please rate how well lectures and/or seminars related to the objectives. Question 4 Constructive written comments on the lectures and/or seminars in this clerkship: (text box) Question 5 Please rate how well chart, bedside or team rounds were conducted. Please rate how patient care opportunities directly applied the facts, concepts and principles presented in the didactic sessions.

32 RESIDENCY PROGRAM DIRECTOR QUESTIONNAIRE Can collect data from the following sources: patient history or history from other relevant information sources (e.g. the family) and physical exams. Has basic science knowledge sufficient to interpret data gathered from the patient interview and physical examination. Knows the sources available for securing additional data through laboratory procedures, and can interpret such data. Can construct a differential diagnosis and a plan for its confirmation or rejection. Treats each patient as an integrated whole, rather than as a disease entity. Uses current literature and other educational resources to understand and resolve diagnostic problems. Can develop multidimensional treatment plans which include diet, place of care, use of medications and therapeutic procedures, and which take into account the patient’s socioeconomic and psychological status.

33 RESIDENCY PROGRAM DIRECTOR QUESTIONNAIRE Learns new concepts in medical literature. Critically evaluates new concepts and interprets new data Respects other health professionals and facilitates the development of their skills. Knows concepts of primary prevention, epidemiology and healthcare delivery and accounts for socioeconomic status. Participates effectively in peer review, especially in providing constructive criticism of professional behavior. Able to choose a career track appropriate to his/her abilities and performance. Has the basics of comprehensive health care delivery, regardless of long-term specialty choice. Is self-directed in solving problems.

34 LEARNING ENVIRONMENT Using internal and external data to improve the learning environment Space Technology

35 CONTINUOUS QUALITY IMPROVEMENT How do you measure success? Set goals Re-measure