1. Formative Evaluation of the Implementation of the Medical Education Metrics Standard for Continuing Education MedBiquitous Annual Conference 2011 May 11, 2011 Baltimore, Maryland

2. Presenters and Disclosures Francis Kwakwa, MA Assistant Director, Data Management, Radiological Society of North America Does not have an interest in selling anything to CME professionals Andrew Rabin, BS Chief Technology Officer, CECity Does have an interest in selling a program or service to CME professionals Sean Hayes, PsyD Vice President of AXDEV Group Does have an interest in selling a program or service to CME professionals Valerie Smothers, MA Deputy Director, MedBiquitous Does not have an interest in selling anything to CME professionals 2

3. Objectives of Today’s Session Increase knowledge about use of Medical Education Metrics Standard (MEMS) in CME Share experience of using MEMS as part of Program and Activity Reporting System Present formative findings collected regarding the experience of those who have implemented MEMS Offer case study findings as to how to assess implementation of new systems 3

4. MedBiquitous Metrics Working Group Mission To develop XML standards & Web services requirements and descriptions for the exchange of aggregate evaluation data and other key metrics for health professions education. 4

5. MEMS: Medical Education Metrics Technology standard for core evaluation data Users –Educators want best practices, ability to compare –Funders want to measure reach and efficacy –Accreditors want to measure success of activity and provider

6. MEMS Data Activity Description –What’s being evaluated Participant Activity Evaluation –What did participants think Participation Metrics –How many people participated Learner Demographics

7. Program and Activity Reporting System (PARS) 7 Manual data entry Comma delimited upload XML upload

8. Formative Evaluation Provides immediate feedback to program participants Fosters reflection and self-regulation Provides feedback about the program itself to designers and sponsors Oriented to improving the focus of the program Based upon a dynamic model

9. Relationship Between Formative and Summative Evaluation Across Life of an Innovation Relative Emphasis Program Life Summative Evaluation Formative Evaluation Worthen 1987

10. Focus Group MEMS implementers using XML in association with PARS 4 organizations represented Measured –Success and satisfaction –Resources for implementation –Characteristics of implementers –Decision factors 10

11. Results Implementers felt satisfied, but limited (not many customers implementing) Limited effort and resources required; change management important Implementers all fluent in XML ACCME adoption key; implementers seeking to better support customers, show competitive advantage, make links between different types of data

12. Case Study Implemented MEMS for PARS upload Experience implementing MedBiquitous standards Integrated new data into user interface and workflow Provided user training

16. Lessons Learned 16 Standards implementation can bring about efficiencies when integrated into workflow and properly supported Validation tools needed Web services needed to realize full benefits, one click data transfer

17. Contact Us 17 vsmothers@jhmi.edu Valerie will forward questions to the working group