1. Curriculum Management – System Best Practices
Robin Rivest, M.Ed., MBA, Director of Curriculum Data Management, Oakland University William Beaumont School of Medicine
Robert Noiva, PhD, Associate Dean for Medical Education Oakland University William Beaumont School of Medicine
Darcy Reed, M.D., M.P.H., Senior Associate Dean for Academic Affairs Mayo Clinic School of Medicine
Terri Cameron, MA, Director, Curriculum Programs, Association of American Medical Colleges
Introductions

2. Disclosures None of the session faculty have anything to disclose.
None of today’s presenters have anything to disclose.

3. Learning Objectives:  
At the conclusion of this session, participants will be able to:
Review features of Curriculum Management Systems
Explain the types of data that are important to the success of a Curriculum Management System
Discuss how Curriculum Management Systems can support curriculum committee discussions, administrative decisions, and student preparation for the curriculum
Share opportunities and challenges from other medical schools
Demonstrate how local curriculum data can be benchmarked with national data
Today you will be reviewing features of curriculum management systems, explaining the types of data that are important to the success of a system, discussing how the system can support committee discussions, administrative decisions, and student preparation, as well as sharing opportunities and challenges from your institutions. Lastly you will see how local curriculum data may be benchmarked to national data.

4. Format of Activities:  
Introduction of Panel Discussion Issues by speakers (30 minutes)
Focus group discussions for session questions (15 minutes)
All groups answer all questions (8 minutes):
What are the best practices for using a curriculum management system?
What are other challenges that curriculum leaders face in documenting their curricula in curriculum management systems for local and benchmarking needs?
Report back to the whole group (5 minutes) (Groups)
Questions/Wrap-up/summary (2 minutes)
Focus Group Questions:
We will come back to these Focus Group Questions later, but you might want to be listening for these themes in the school presentations and see how that information can help in your small group discussions.

5. Mayo Clinic School of Medicine

6. MCSoM Curriculum Management System
Evaluation
Improvement
Implementation

7. MCSoM Curriculum Management System
Evaluation
Improvement
Implementation
Student Competency
milestones
EPAs
Student Satisfaction
Faculty Performance
Curriculum Inventory
Content
Structure
Delivery
Evaluation
Program Outcomes

8. MCSoM Curriculum Management System
Evaluation
Improvement
Implementation
Accountability
Students
Faculty
Leadership
LCME
Patients
Community/ Public
Dissemination
Lessons learned
Best practices
Scholarship

9. Curriculum inventory: most important tool in the CM toolbox
5 Practical Ways the Curriculum Inventory helps the Office of Academic Affairs
… reflections from one curriculum associate dean

10. 1. Where do we teach it? Find any topic within the curriculum
Requires comprehensive, granular data entry
Identify objectives & corresponding assessments
Respond comprehensively to queries from leadership, faculty & other stakeholders
Explain & justify decisions regarding curriculum changes
Advocate for resources in priority areas

11. 2. Who is our expert? Quick way to identify faculty who teach
specific content & what teaching &
assessment methods are used

12. 3. How do we measure up? National benchmarks
Compare performance to others
Identify strengths & opportunities for growth
In what ways are we unique?
What are we doing that is
innovative?

13. 4. Accreditation preparation & continuous monitoring
Inventory reports relevant to multiple standards and elements
Extremely helpful to complete several sections of the DCI
Facilitates self-study
Expedites identification of gaps, deficiencies

14. 5. I wonder if…
Regular review of curriculum data within curriculum committees & during annual course reviews, and with course directors & clerkship directors stimulates research questions
Reports provide baseline data
Support faculty in education scholarship

15. Curriculum Management: Lessons Learned
Review data at every opportunity
Auto-populate all committee agendas
Develop familiarity, identify issues, trends
Make it easy for faculty
Big picture first, details second
Data verification role
Quick access to reports for all stakeholders
Transparency
Highlight changes, improvements, progress

16. Introduction Slide:
OUWB School of Medicine is one of a growing group of new medical schools. We are located in Rochester, Michigan, a suburb of Detroit. We have a 2+2 curriculum using an integrated, organ-based, pre-clinical curriculum with traditional clerkships. We have a single clinical partner, Beaumont Health, with 8 hospitals serving southeast Michigan. We received full accreditation in February 2015 and graduated our first class in May 2015.

17. OUWB Curriculum Management System
Curriculum Inventory
Data documenting the content, delivery, and assessment of an appropriate curriculum
We include student assessment, faculty and course evaluation, and faculty workload in our inventory
Curriculum Management System
Collection of data
Archiving and access of data
Internal and external reporting of data
How do we define our data management system?
Understanding our goal is to do the best job possible of educating competent, inquisitive, compassionate, and ethical student physicians, the curriculum is a key component of that process.
We must ensure that the curriculum addresses each of our identified competencies for graduation.
The curriculum inventory provides us with the data to ensure we are meeting our educational mission. The system requires that we
1) Collect appropriate data
Archive that data in a protected environment that allows adequate access for our review, and finally
Generate reports for both internal and external requests for review of that data
Because of the power of the software we have purchased to provide this data, we also have the ability to collect data that documents student assessment, faculty and course evaluation, and faculty workload in our inventory. These data points have been useful in ensuring we are meeting our educational responsibilities and outcomes goals and fulfilling our mission.

18. OUWB Curriculum Management System
People
Office of Medical Education (Director of Curriculum Data Management)
Curriculum Committee and Curriculum Integration Subcommittee
Information Technology (library faculty)
Process
Data entry, reporting, review
Data on content, delivery, assessment, workload
Tools and Resources
OASIS, Moodle, iSeek
Our goal for this session is to share some thoughts regarding the management of our curricular content and how a young medical school continues to refine the process of data collection, reporting, and evaluation. Let me break down the data management system in terms of people, process, and tools and resources.
Different schools identify the responsibilities for curriculum data management using a variety of strategies. At OUWB we share the responsibility between the OME, faculty committees, and library staff. I, as the Director of Curriculum Data Management, am the single dedicated staff member supporting the software and databases. I oversee the process of entering, reporting, and reviewing the content, delivery, assessment, and workload data using a collaborative process of the three groups, delivering reports to administration, staff and faculty committees. We have chosen OASIS, Moodle, and iSEEK as our primary tools for data entry, archiving, and reporting.

19. OUWB Curriculum Management System
What challenges were identified?
What are our opportunities for improvement?
What should be best practices?
Now that we’ve successfully defined a process and have uploaded 4 years of curriculum inventory data to the AAMC, we are reflecting on our practices, identifying challenges, opportunities for improvement, and our goals for best practices in the future.

20. Curriculum Management System PDSA Cycle
Assess
Design
Develop
Implement
Evaluate
Improve
Curriculum Data
Management
The curriculum inventory management process was implemented 4 years ago. The data generated has been instrumental in the evaluation and continuous improvement of our curriculum. But what about the curriculum data management process itself?
Version 2.0

21. OUWB Curriculum Management System People
Challenges
Opportunities
Best Practices
Decentralize roles in the process with varying levels of responsibility, reliability, and expertise
Maintain stakeholder engagement
Meet the individual needs of all of the data users
Seek “fresh eyes” integrating new leadership and faculty
Evolve committee members and practices
Leverage the expectations of professional organizations
Establish positive relationships
Provide content to edit rather than expect faculty to generate data
Collaborate interprofessionally
Utilize regular face-to-face meetings to foster collegiality
Let’s begin with the people.
Our challenges began with the realization that we needed to improve data input and verification from faculty. The expertise of faculty on content and quality of reported data was essential. The roles of participants in the process were decentralized, therefore, recognizing people needed to have varying levels of responsibility, reliability, and even expertise in the managing of curriculum data. This resulted in improving stakeholder engagement while at the same time easing their burden. The challenge required us to identify and meet the unique, individual needs of all of the data users.
Some of our people challenges include the reality that we have users entering data at different points in time. Like most schools we are continually integrating new course and clerkship faculty and leadership personnel and faculty. Although this is a continuing professional development challenge, it also gives us new perspectives when we have “fresh eyes” to comment of the usefulness of the data and reports. This also provides us with opportunities to evolve our committee memberships and leverage expectations that professional organizations are sharing with their members to help us determine any gaps in curricular content.
It’s important for us to build on our best practices and those include the building of positive relationships with the curriculum team with faculty and other School of Medicine stakeholders such as student affairs, IT, and our communications office.
Finally, an important ethos of our group is that we are service providers, providing content and support to staff and faculty. We have adopted a collaborative strategy of asking faculty to edit data rather than generate the data on their own. We also work to collaborate interprofessionally with clinical staff, university personnel, and internally as a team, using regular face-to-face meetings to facilitate the process and improve collegiality.

22. OUWB Curriculum Management System Process
Challenges
Opportunities
Best Practices
Simplify identification of gaps in content and assessment
Deal with synonyms
Need to resolve redundant data in the map (ex: repetitive instruction)
Audit for inaccurate data (ex: faculty workload)
Redesign reports to meet committee needs
Improve topics and keyword lists
Audit data collection and validate data
Ensure timely and accurate data for ALL learning (electives, rounding, online modules, textbooks)
Medbiquitous terminology
Identify mapping strategies for 100% of the curriculum, including self-directed learning and co-curricular
Meet additional LCME requirements
Continuously evaluate and improve the process (PDSA cycle)
Next let’s look at the process.
One thing we noticed was that identifying gaps in content and assessment gaps was difficult because of terminology and keyword limitations and how we are filtering our searches. Synonyms were a problem. Is it BPH, benign prostatic hyperplasia, or a Prostatic Adenoma, or Prostatic Neoplasms? We have also struggled with redundant data that occurs in the inventory because of platooned instructional sessions or other instances of repetitive instruction. We didn’t want to double count hours. We also identified the need to audit all data during the process. Otherwise we were missing information about unplanned session and faculty changes.
The PDSA cycle has lead to identified opportunities to redesign reports for our various committees, improve our keywords list, audit and validate data, and be more inclusive of our data, all based on stakeholder feedback from both individuals and committees.
As we move forward we would like to identify synonyms and use Medbiquitous terminology to better standardize the process. Like most schools we struggle with how granular the data should be. Our stakeholders have noted a gap in the inventory in the area of self-directed learning, especially in the clerkships. There is a lot of content that students are expected to master, that needs to be more thoroughly mapped in instructional sessions. We are exploring ways to improve the inclusion of rounding, online modules, and textbook objectives into our data.
We have also implemented a parallel mapping system that tracks competencies learned in co-curricular activities.

23. OUWB Curriculum Management System Tools and Reports
Challenges
Opportunities
Best Practices
Refine the dataset
Accessibility/Protection
Accuracy
Timeliness
Improve reports
Usability
Quality
Implement change control
Improve linkage of curriculum content to student assessment
Provide better data for curriculum support and accreditation
Correlate curriculum with program outcomes
Demonstrate achievement of each student competency
Benchmark program content and outcomes with AAMC Curriculum Inventory Reports
Develop dashboards for reporting of program outcomes success
Produce curriculum integration topic reports in a timely manner
Next let’s look at the tools and reports.
The quality of the reports is obviously determined by the accuracy of the data within them. Therefore, we recognized early on that the dataset needs to be refined on a regular basis to ensure for accessibility and protection, as well as the accuracy and timeliness of the data in the database. Our goal was always to generate accurate reports in a timely manner to meet the needs of the stakeholders. We have many different faculty, administrators, and coordinators all accessing and using the data at various points in time with varying needs and levels of IT expertise. And there lies the challenge.
Some areas we are looking to improve on include our ability to measure how we use the inventory to evaluate outcomes. Leadership wanted dashboards. The dashboards are data reports that make it easy to benchmark outcomes internally between programs and compare our programs nationally. Most importantly, the dashboards show us where we are meeting our expectations of educating competent physicians and also meeting our established program competencies.
Our mandate from the beginning was always to provide faculty with the data required to demonstrate student competency and measure program outcomes. We all know that the LCME accreditation process is a driving force, but our faculty has all believed we needed to practice evidence-based education. They wanted us to provide them with the reports that would clearly identify whether they were successful. It was our responsibility to identify successes and failures with accurate and timely data. We have been able to provide them with those reports. Our curriculum inventory clearly demonstrates our instruction and assessment of each student competency. We are also able to provide faculty with benchmark reports comparing ourselves against national data in many areas. This has been quite useful.
Effective reporting remains a goal. Faculty and administrative leadership appreciates reporting dashboards that simply display program outcome success.
Our greatest success has been the creating of on-demand curriculum integration topic reports that provide faculty and leaders with curricular content in a timely manner. Soon I will show you an example of one that our faculty and committees find very useful.

24. Examples of overall successes and challenges identified using the PDSA Cycle
Success – only minor modification
Challenge – needed redesign of inventory process
identifying competency content coverage
generating topic content reports
benchmarking outcomes
reporting faculty instructional workload
inventorying self-directed learning (clerkships)
unreported clerkship content, ex. genetics
Following the PDSA cycle helped us to identify our successes and challenges.
As examples, we have successes that have only required minor process changes and other challenges that have required greater attention.
End here
We are able to identify topic coverage in our content reports and use that knowledge to see how we compare in our coverage to other medical schools, which is important for a new medical school.
We face challenges when it comes to meeting data requirements to support curriculum inventory and also workload, attendance, etc. The way the data is sliced and diced and how well the data structure in place can best support those requirements is a challenge. Following the cycle lead us to the conclusion that we need to input more data, particularly in the clerkships where learning experiences are harder to quantify in terms of hours and numbers. Having the ability to drill down from topics into types of topics is needed. For example, when looking at where and how genetics is taught in our curriculum, it becomes apparent that you need to specify if you’re looking for genomic competencies, or any and everything related to genetics, genetics counseling, for example.

25. Samples of the Work Benchmarking Report
Weeks of instruction
Contact hours
Types of instruction
Types of assessment
Sequencing
Graduation requirements
We would next like to share some of the sample reports we provide to our stakeholders.
First, the AAMC Discipline Benchmarking Reports have been quite valuable to faculty, committees, and course leaders. This type of report gives faculty a comparison of our programs to other schools. For example, we have compared our inventory data with national data to analyze weeks of instruction, contact hours, types of instruction and assessment, as well as sequencing and graduation requirements. This report is useful for reviewing the horizontal and vertical integration of the curriculum.

26. Samples of the Work Content Report
Topic Reports: Abuse content in the curriculum
Request driven by faculty/leadership/Provost Office proposal
Data collection process involves Office of Medical Education and Curriculum Integration subcommittee
Content of report
Instruction on Abuse in the OUWB Curriculum
Abuse curriculum benchmarked at other medical schools
Bibliography of literature on abuse curricula in Medical and Health Sciences education
Another example is what we call ad hoc topic content reports.
One of the responsibilities of our Curriculum Committee is to evaluate the content of the curriculum regularly. These reports allow us to request an inventory of any curriculum topic and how it is integrated throughout the four years of instruction.
Committee members, leadership, and faculty regularly approach us and ask us “Where is….. (insert some topic) covered in the curriculum?” It might be an orphaned topic like “healthcare economics” or a very broad topic like “ genetics or nutrition”. The Ad hoc Topic Content Report provides faculty with a summary of how any content area is currently represented in our curriculum.
This report includes a calendar of sessions and course and clerkship objectives associated with the issues covered within the topic. Specific courses and sessions are identified as well as what instructor taught them. Hours, session numbers, and sequence in the curriculum are identified. In addition, an overview of how the content area is taught and assessed at other medical schools is included through a brief literature review and bibliography. The report is always reviewed by a committee member with expertise before dissemination to improve it’s accuracy and usefulness.

27. Curriculum Mapping Analysis Competency Domains
Another example of a report that was embraced by leadership and the curriculum committees was the curriculum mapping analysis report which charts the mapping of the competency domains and the individual competencies.
For simplification and readability, I have included our mapping of the 8 PCRS competency domains without the mapping of each individual competency.
Although it is simply a frequency report, this report provided users with data indicating how much content mapped to each competency. We were able to identify competencies where there may have been less than 5 instructional sessions, activities, or assessments that mapped to any specific PCRS competency. Course and clerkship leadership and curriculum committees could very easily identify areas of the curriculum where there might be possibilities we were not covering content or skills adequately. Staff could also provide what those sessions, activities, and assessment were. In some cases, the analysis also pointed out that we weren’t adequately documenting all of the content. This lead to a review of the processes for documenting curricular content.

28. OUWB Curriculum Management System
Keys to Continuous Improvement of the Curriculum Management System
Periodically evaluate your curricular data management system, getting feedback from users and stakeholders
Recall your initial curricular goals when evaluating the success of your curriculum management system
Use the service provider concept to meet the need of your stakeholders
Like you, we are always seeking new ways to continuously improve our curriculum management system.
As a summary, we have included some of our key concepts in ensuring the success of our own curriculum inventory process.
We hope that in sharing our challenges, opportunities, and best practices with you, we may have given you some useful ideas to consider as you evaluate the curriculum management system at your own school.

29. Discussion Questions: 15 Minutes: 4 Mins per Question; 5 Min Wrap-Up
What are the best practices for choosing a curriculum management system?
What are other challenges that curriculum leaders face in documenting their curricula in curriculum management systems for local and benchmarking needs?
Please divide into groups and discuss these two questions. We would like for you to be able to discuss the questions and then do a quick wrap-up of the summary you will present from your group. We would like the scribe for each group to either give hand-written notes to one of the session faculty or them to within a week. We’ll leave the questions on the screen during the discussion.

30. Benchmarking Local Curriculum Data with National Data
Terri Cameron, MA, Association of American Medical Colleges
Walter Fitz-William, MPP, Association of American Medical Colleges

35. Session Wrap-Up What are some ‘take-home’ messages from this session?
What action items will you take back to your institution?
What resources do you need to assist you with this process?
What colleagues have you just met who could work with you to implement new strategies or help you with resources?
While these sessions are always great for learning new information and processes, as well as for meeting new colleagues, we are exposed to so much information at these meetings that we sometimes lose great plans and intentions by the time we return to our offices. Let’s take a few moments to organize our thoughts and make meaningful use of what we have just learned. And, please remember to give your group notes to one of the session faculty or them to