Educational Scholars Program PAS: May 6, 2008 Academic Pediatric Association

2 CME Disclosures The speakers at this session have no relevant financial interests to disclose or conflicts of interest to report. The speakers at this session have no relevant financial interests to disclose or conflicts of interest to report.

Mid-course monitoring of an educational program: What are the crucial sources of information?What are the crucial sources of information? How should one respond to these data?How should one respond to these data?

4 How would you apply these guiding principles?  Planned curriculum: 14 learning goals  Principles of adult learning in all interactions Curriculum reflects identified needs of learners Hands-on practice of authentic tasks Scholarly projects with mentoring and regular feedback Developmental evaluations of authentic educator documents  Professional networking and peer learning  Dynamic, evolving learning program, shaped by needs assessments, evaluation and feedback from participants  Serious research and evaluation of program to help build sustainable support

5 How would you apply needs assessment results?

6 Needs Data: See Handout  Cohort I: Years 1 vs 2 Items with greatest and least reported increase in competence  Cohort 2 Start of Year 1 Items with greatest and least reported competence

7 Our Response to the Needs Assessment Results  Modified curriculum each year to address highest perceived needs: Yr 1: Evaluation, qualitative research (ISMs and PAS session) Yr 2: Education technology and problem learners (ISMs) Yr 3: Peer feedback (scholarly writing wasn’t included in the needs assessment)

How would you apply program evaluation results?

9 Program Evaluation by Scholars ESP ActivityTime Expended (hrs) Interest % >3 Value % >3 PAS sessions (mean of 2) %47% Intersession Modules (Mean of 4) %37% PAS Workshop reviews %36 % Developing EP %63 % Scale 1=low 5=high

10 ESP ActivityMean Time Expended (hrs) Interest >3 Value % >3 Conducting your project 6388 %84 % Writing project report 736 %38 % Interactions with mentor (Not ESP) 1648 %50 % Interactions with ESP facilitator re project 448 %35 % Interactions with ESP facil re EP/CV/career 354 %46 % Program Evaluation by Scholars Scale 1=low 5=high

11 Overall Ratings of ESP by Scholars Effort expended 5.9 Value received 6.2 Scale 1=low 9= very high

12 Our Response to the Program Evaluation Results  Modified timeline of assignments  Reduced number of Intersession Modules  Continued strong emphasis on project reports and EPs  Considering reducing workshop review requirement  Hope to increase faculty interactions with scholars!

13 How can you use an educational program as a springboard for educational scholarship ?

14 Rationale for the EP Research Project  ESP Scholars needed EP for career development  We needed EPs for scholar and program evaluation  We needed research opportunities to reward our volunteer faculty RESULT: The EP Project

15 Quick Project Overview Since Jan 2006  Developed EP template based on findings of AAMC national consensus conference  MedEdPortal review and approval  Construction of EP analysis tool by consensus development process, using Scholars’ EPs as raw material  Revision of template to match tool  Presentation to AAMC-GEA Steering Comm  Article submitted to Academic Medicine  Multiple workshops on EPs

EP Template List of >100 QuantitativeItems List of 52 QualitativeItems Tool 1.1: Tool 1.1: Selected and Combined 43 Items Tool 1.2: Refined and ReconciledItems Tool 1.3: Final content andFormat MedEdPortalApproval MedEdPortalReview EP Test 4 n=2 8 raters Development of the Educator Portfolio Analysis Tool EP Test 3 n=3 5 raters EP Test 5 n= raters Tool Development Inter-rater Reliability Testing EP Template Revision EP Test 1 n=27 6 raters EP Test 2 n=5 4 raters EP Template Revision Template Development See handout for item list

17 Future Plans  Interrater reliability testing with new EPs (with article)  National consensus development process through AAMC  National dissemination via workshops  EP website with electronic data entry/analysis ($$ to support ESP)  National norming study

18 ESP Business

19 Review of Program Requirements: Cohort 1  2 workshop reviews/yr (due at ESP session)  Intersession Module 1 by Oct 15  Final Educator Portfolio and updated Curriculum Vitae by Jan 15, 2009  Peer review of another scholar’s EP by Mar 15 (substitutes for Intersession Module 2)  Evidence of peer reviewed acceptance of a ESP project publication or national presentation by Dec 2009 or thereafter  Final project report at program completion  We will ask for EPs and CVs periodically in the future

20 Cohort 2: Simplified ESP Assignment Timeline  At PAS: 2 workshop reviews  July 15: Annual Project Reports  Aug 30: Program evaluation survey  Oct 15: Intersession Module 1 (technology in educ)  Jan 15: Updated Educator Portfolios and CVs  Mar 15: Intersession Module 2 (EP peer review)

21 Agenda for Today  8:00-8:30 Introduction/orientation  8:30-12:00 Turning Scholarly Projects Into Scholarly Products 8: :00 Writing educational scholarship articles 10: :00 Planning and presenting workshops 11: :00 Preparing and giving platform presentations  12:00- 1:30 Lunch and discussion: Cohort 1: How to take lessons learned from ESP to move forward in one’s career? Cohort 2: Discussion of projects  1:30-4:15 Developing Peer Feedback Skills 1:30-1:45 Giving and receiving peer feedback 1:45-2:45 Platform session with feedback 2:45-3:45 Poster review with feedback 3:45-4:15 Feedback discussion and ESP session evals  4:15-5:00 Feedback from Scholars on the ESP program Break