1. Developing a Summer Research Program for Medical Students: From Development to Evaluation J. Ferrante, S. Rovi, C. Brazeau, M. Vega, P. Chen, L. Boyd UMDNJ-New Jersey Medical School Newark, New Jersey

2. Objectives Modified due to time constraints  Obtaining funding  Selection of research projects  Recruitment of students  Formal curriculum  Evaluation  See handout for information on these objectives

3. Background Shortage of primary care researchers Shortage of primary care researchers Need to train future primary care researchers Need to train future primary care researchers Exposure to Family Medicine research mentors Exposure to Family Medicine research mentors Lack of non-bench research opportunities Lack of non-bench research opportunities Increase the department’s research productivity Increase the department’s research productivity

4. Summer Research Program 8-week program 8-week program Students recruited and selected to work on faculty’s projects Students recruited and selected to work on faculty’s projects Formal research curriculum Formal research curriculum Opportunity to shadow physician Opportunity to shadow physician

5. Curriculum for 8-week Summer Research Program Prior to summer Prior to summer  Completion of mandatory Human Subjects Protection course 1 st week 1 st week  Daily didactic sessions with faculty each morning  Individual meeting with the sponsor faculty each afternoon

6. Curriculum for 8-week Summer Research Program Weeks 2 –7 Weeks 2 –7  Weekly didactic sessions  Student sharing of project process  Meetings with faculty sponsor as negotiated Week 8 Week 8  Presentations of research projects by students  Poster presentation  Submission of abstract at end of summer

7. Aim of curriculum To promote students’ interest in primary care research To promote students’ interest in primary care research To develop students’ knowledge in primary care research To develop students’ knowledge in primary care research To enhance students’ skills in primary care research To enhance students’ skills in primary care research To promote mentorship with family physician To promote mentorship with family physician

8. Curriculum Objectives Students will understand: Importance of doing research in Family Medicine Importance of doing research in Family Medicine Principles of study design in primary care research Principles of study design in primary care research Tools for finding research information Tools for finding research information Responsible conduct of research Responsible conduct of research Managing data and significance of statistical analysis in primary care research Managing data and significance of statistical analysis in primary care research Presentation of scientific results Presentation of scientific results

9. Evaluation of Summer Research Program (SRP)  Curricular topics for SRP (2003-2005)  Student Satisfaction with SRP (2003-2005)  Pre and Post Assessment of Student’s Research Knowledge, Attitudes and Practice (KAP) (2004-2005)  Anonymous written comments & facilitated discussion of SRP Number of students participating in SRP for all 3 years: 22

10. Evaluation of Curriculum Students anonymously scored 10-12 didactic sessions provided each summer. Students anonymously scored 10-12 didactic sessions provided each summer. Overall mean score: 3.55 (0.35) Overall mean score: 3.55 (0.35)  Scale: 1=“not very informative” to 4=“very informative” 4=“very informative” Median=3.63 (2.67 - 4.00) Median=3.63 (2.67 - 4.00)

11. Student Satisfaction with SRP Students anonymously rated 10 items of the SRP on satisfaction Students anonymously rated 10 items of the SRP on satisfaction Overall mean score: 4.57 (0.38) Overall mean score: 4.57 (0.38)  Scale: 1=“not very satisfied” to 5=“very satisfied” 5=“very satisfied” Median=4.67 (3.67 -5.00) Median=4.67 (3.67 -5.00)

12. Satisfaction with program ASPECTS OF PROGRAM SCORE Curriculum overall4.6 Coordination of the program4.6 Faculty supervision4.8 Interactions with other DFM faculty4.5 Facilitation of informal weekly discussions4.3 Shadowing experience4.7 Your designated role on the research project4.3 Preparation for student presentations4.8 Student presentations (i.e., doing/hearing presentations) 4.8 Overall satisfaction4.8

13. Pre & Post Surveys on Research Knowledge, Attitudes & Practices (KAP)*  Administered in 2004 & 2005 (N=16)  29 items : LOW (1), MODERATE (2) or HIGH (3)  Knowledge  Willingness to Engage (Attitudes)  Performance (Skills)  Sample items  Formulating a research question  Developing a hypothesis  Recruiting patients  Writing an abstract * Kleber, E. Self Report of Research Skills and Knowledge of Health * Kleber, E. Self Report of Research Skills and Knowledge of Health Professionals. New Jersey Nurse. 2002; 32(6):11. Professionals. New Jersey Nurse. 2002; 32(6):11.

14. Pre and Post KAP Surveys Overall Mean Scores PRE & POST SRP PREPOSTp-value Knowledge1.8(0.32)2.3(0.27)0.037 Attitudes2.6(0.34)2.7(0.22)0.238 Practices/Skills1.8(0.50)2.4(0.41)0.008

15. Selected Students’ Comments “I was very satisfied with the summer program in every aspect.” “I was very satisfied with the summer program in every aspect.” “The weekly meetings were very effective because they were interactive.” “The weekly meetings were very effective because they were interactive.” “I liked the freedom I was provided while working on my project and that my input was respected.” “I liked the freedom I was provided while working on my project and that my input was respected.”

16. Summary of Student Evaluations of the SRP  Didactic sessions: “very informative”  Satisfaction: “very satisfied”  Research Knowledge and Practice/Skills increased significantly from Pre-SRP to Post-SRP  ANOVA comparisons by year for all evaluations showed similar results  Students comments were positive overall and helpful in making modifications to the SRP

17. Benefits of program Build relationships with students Build relationships with students Expand research reputation of Department Expand research reputation of Department Gain valuable assistance with small projects Gain valuable assistance with small projects  Great source for pilot data and grant applications Incentive to complete project Incentive to complete project Publications and presentations from studies Publications and presentations from studies

18. Challenges of Program Time intensive for research faculty and project faculty Time intensive for research faculty and project faculty Space restrictions Space restrictions Comparison of different projects Comparison of different projects Extra funding to support enough students to make it worthwhile Extra funding to support enough students to make it worthwhile

19. Acknowledgments BrHP/HRSA 1 D16 HP 00141-01 BrHP/HRSA 1 D16 HP 00141-01 UMDNJ- New Jersey Medical School Summer Research Program UMDNJ- New Jersey Medical School Summer Research Program AAFP Foundation G0309 AAFP Foundation G0309 AAFP Foundation G0211RS AAFP Foundation G0211RS Department of Family Medicine, UMDNJ-NJMS, Newark Department of Family Medicine, UMDNJ-NJMS, Newark

20. Contacts Jeanne Ferrante, MD Jeanne Ferrante, MD  ferranjm@umdnj.edu ferranjm@umdnj.edu Sue Rovi, PhD Sue Rovi, PhD  rovisl@umdnj.edu rovisl@umdnj.edu

21. Thank you

22. Objectives and other material not presented due to time constraints. Obtaining funding Obtaining funding Selection of Research Topics Selection of Research Topics Recruitment and Selection of Students Recruitment and Selection of Students Formal curriculum: Design, Workshops Formal curriculum: Design, Workshops Sample presentations and publications Sample presentations and publications

23. Obtaining Funding Multiple funding sources key Multiple funding sources key UMDNJ summer research program: UMDNJ summer research program: Yearly, competitive Faculty applies, research office selects the projects NJAFP/AAFP joint summer program: Yearly, competitive NJAFP/AAFP joint summer program: Yearly, competitive Student applies, NJAFP selects the projects

24. Obtaining Funding SHARE SHARE Community service, research mentor BHPr/HRSA grant: BHPr/HRSA grant: Grant obtained for 3 years, covering part of student stipend (due to federal limits) Family medicine department covers rest of stipends and selects the projects Funded educational research projects

25. Selection of Research Projects Limited time to complete individual project Limited time to complete individual project Students work on faculty’s projects Students work on faculty’s projects Faculty submit internal proposal Faculty submit internal proposal Same as school’s application Same as school’s application

26. Selection of Research Projects Research committee reviews proposals Research committee reviews proposals  Scientific merit  Role of student  Maximize research funding  Number of positions available  Number of sites

27. Examples of Projects Rates of Breast and Cervical Cancer Screening in Obese Women Rates of Breast and Cervical Cancer Screening in Obese Women The Medical Management of Sexual Assault Victims in the Emergency Department The Medical Management of Sexual Assault Victims in the Emergency Department Improving Identification of Child Abuse & Neglect Through Analysis of Associated Diagnoses Using National Hospitalization Data Improving Identification of Child Abuse & Neglect Through Analysis of Associated Diagnoses Using National Hospitalization Data Evidence Based Medicine: A Survey of NJ Physicians’ Practices Evidence Based Medicine: A Survey of NJ Physicians’ Practices

28. Examples of Projects Readability of Cancer Related Patient Education Materials Readability of Cancer Related Patient Education Materials Primary Care Physician’s Knowledge and Practices Regarding Chronic Hepatitis Primary Care Physician’s Knowledge and Practices Regarding Chronic Hepatitis Domestic Violence in Family Practice Settings: Domestic Violence in Family Practice Settings: A Comparison of 3 Screening Protocols Multimethod Assessment of Preceptor Feedback and Teaching Style during the Family Medicine Clerkship Multimethod Assessment of Preceptor Feedback and Teaching Style during the Family Medicine Clerkship

29. Recruitment Family Medicine funded projects Family Medicine funded projects  Email to students  Flyers in 1st years’ mailboxes  In-class announcements School-funded projects also used School-funded projects also used  Website listing of projects  On-line application form  Group orientation meeting with students Yield: 30 applications for 8 positions this year Yield: 30 applications for 8 positions this year

30. Recruitment Unique to Family Medicine Projects Unique to Family Medicine Projects  Formal research curriculum  Shadowing a physician  Invited to work with a faculty physician at least once during the project

31. Selection of Students Students contact individual faculty Students contact individual faculty Faculty interview students Faculty interview students Preference given for students interested in Family Medicine or primary care careers Preference given for students interested in Family Medicine or primary care careers Interest in particular subject matter Interest in particular subject matter

32. Process of Curriculum Design DFM research committee meetings to discuss structure and content of curriculum DFM research committee meetings to discuss structure and content of curriculum Scheduling of faculty to present workshops Scheduling of faculty to present workshops

33. Workshops – Week 1 Bagels and donuts: Introductions and overview Bagels and donuts: Introductions and overview Importance of doing research in FM Importance of doing research in FM Overview of HRSA grant Overview of HRSA grant Who’s doing what: Discussion of individual projects Who’s doing what: Discussion of individual projects Reviewing the Literature Reviewing the Literature

34. Workshops – Week 1 Research questions and research designs Research questions and research designs IRB issues: 10 questions led to discussion IRB issues: 10 questions led to discussion Evidence-Based Medicine Evidence-Based Medicine Dealing with data: Data abstraction, missing data, data cleaning, data coding, variable creation, recoding, etc. Dealing with data: Data abstraction, missing data, data cleaning, data coding, variable creation, recoding, etc.

35. Workshops – Weeks 2-8 Survey research Survey research Focus group research Focus group research Statistical analysis Statistical analysis Writing an abstract Writing an abstract Preparing an oral presentation Preparing an oral presentation Preparing a poster and PowerPoint presentation Preparing a poster and PowerPoint presentation Presentations by students with faculty invited Presentations by students with faculty invited

36. Sample Presentations National Centers of Excellence in Women’s Health 2003: Chen, Rovi, Vega, Jacobs, Johnson. “Health Status Differences between Victims of Domestic Violence and Non-Victims in Primary Care Settings.” National Centers of Excellence in Women’s Health 2003: Chen, Rovi, Vega, Jacobs, Johnson. “Health Status Differences between Victims of Domestic Violence and Non-Victims in Primary Care Settings.” NYS Regional Society for Academic Medicine 2003: Rajwani, Scarpone, Rovi, Atkin. “Provision of prophylaxis to rape victims in an Emergency Department.” NYS Regional Society for Academic Medicine 2003: Rajwani, Scarpone, Rovi, Atkin. “Provision of prophylaxis to rape victims in an Emergency Department.” Victimization of Children and Youth International Research Conference 2004: Rovi, Burrows, Chen, Johnson. “Diagnoses Associated with Hospitalizations of Child Sexual Abuse.” Victimization of Children and Youth International Research Conference 2004: Rovi, Burrows, Chen, Johnson. “Diagnoses Associated with Hospitalizations of Child Sexual Abuse.” WONCA 2004: Ferrante, Chen, Jacobs. “Breast and Cervical Cancer Screening in Obese Minority Women.” WONCA 2004: Ferrante, Chen, Jacobs. “Breast and Cervical Cancer Screening in Obese Minority Women.”