Joseph Brocato, PhD, Erik Solberg, MA, and Shailendra Prasad, MD, MPH Department of Family Medicine and Community Health Locus of Control and Self-Assessment Measures as Predictors of Family Medicine Resident’s Academic Performance: Year Two Cohort Results

Presentation Objectives –Define locus of control (LOC) and self-assessment (SA) and their potential relationship to academic performance –State the research design used for a study of LOC and SE at the UMN Dept of Family Medicine and Community Health –Identify the elements of the instruments to measure LOC & SA –Describe the next steps in the five-year study of UMN Family Medicine Resident’s LOC & SA

Background Issues Decisions made during residency selection have long-lasting implications FM Residency education is not uniform No standard experiential component to determine clinical exposure Breadth of training in Family Medicine is expansive and expanding

Global Challenges Residents coming in from diverse background and different learning styles Patients don’t check board scores Current academic measures may not be indicative of future academic performance Ill-defined, inconsistently applied, non-existent residency selection criteria

Current Methods of Resident Selection Limited mutual exposure/surface level assessment of candidates Traditional application materials (transcripts, boards, personal statement, letters of recommendation) Brief interview with PD, Residents, Faculty Program Extracurriculars (team fit): dinners, didactics Much rarer: observerships, clinical rotations

Current methods of measuring a resident’s performance In training examination (ITE) Board Scores Rotation based evaluations of medical knowledge Other evaluations –Structured & Unstructured

Key Concept: Locus of Control Originally described by Rotter in 1950s Individual’s perception of underlying causes of events in life Conceptualized as a unidimensional continuum from an internal to an external

What do we know about LOC and physicians LOC may influence communication skills training. Belgian study- Libert et al Self-confidence and physical exam- Wu et al

Internal LOC “Actions are within ones control” General acceptance that internal LOC is desirable Studies in the 1980s that less people higher internal LOC going into primary care

External LOC “Actions guided by outside factors” Better for teamwork ? Could be learned and culturally determined Is this bad?

Key Concept: Self Assessment Y. Trope, (1982, 1975) –Model to predict task selection –Self assessment is narrowly predictive: it is limited to specific conditions –Based on the premise people strive to attain a realistic understanding of weaknesses and strengths to better predict and effectively cope with their environment –The attractiveness of a task depends on the extent to which it can reduce uncertainty about one’s ability level –All outcomes can have positive values. The scale of a success or failure determines the net impact on the individual

Research Questions from our Study What is the LOC & SA of FM residents? Change over time? Differences in LOC & SA by demographic variables? Are LOC & SA strong predictors of traditional academic performance during residency?

Research Design: Longitudinal Cohorts Five years in study length (allows for n=120) –Study Year One: Cohort 1 (PGY 1) –Study Year Two: Cohort 1 (PGY 2), Cohort 2 (PGY 1) –Study Year Three: Cohort 1 (PGY 3), Cohort 2 (PGY 2), Cohort 3 (PGY 1) –Study Year Four: Cohort 2 (PGY 3), Cohort 3 (PGY 2) –Study Year Five: Cohort 3 (PGY 3) Non-matriculating residents will be dropped from the study Transferring residents not be included

Research Design: Overview & Timeline 5 year study LOC & SA assessment –PGY 1: During first month & during last month of internship/PGY 1 –PGY 2: During last month of PGY 2 –PGY 3: During last month of PGY 3 Resident Academic Performance –In Residency: Board Scores, ITE Exams –Post-residency: Board Certification Exams

About our residents There are a total of 55 first year residents (166 total) within eight sites across MN Residents from the Duluth program (10 residents) did not participate in the study as they are a new residency in the system.

Resident sample demographics: Cohort 1 33 of 48 agreed to participate in the study (69%) 33 residents completed the locus of control survey (33/33=100%)* –27 MDs, 6 DOs –14 men and 19 women –Includes 12 IMG residents 4 men and 8 women »*During year 2, one resident opted out of the study and one has not completed the instrument for a total response rate of 31/33 (94%) 29 residents completed the self-assessment survey (29/33=88%) –23 MDs, 6 DOs –10 men and 19 women –Includes 12 IMG residents 4 men and 8 women

Resident sample demographics: Cohort 2 39 of 45 agreed to participate in the study (87%) 37 residents completed the locus of control instrument (37/40=95%) –31 MDs, 5 DOs, 1 MBBS –20 women and 17 men –Includes 6 IMG residents 4 men and 2 women 36 residents completed the self-assessment instrument (36/39=92%) –30 MDs, 5DOs, 1 MBBS –19 women and 17 men –Includes 6 IMG residents 4 men and 2 women

Research Design: Instrumentation Survey Monkey Longitudinal Resident Self-Assessments –Resident Self-Assessment of Knowledge and Skills –Duttweiler Internal Locus of Control Index

Resident Self Assessment of Knowledge and Skills Demographics &Tracking information (confidential, not anonymous) Knowledge areas based on FM program requirement subject matter Scale for knowledge base based upon Bloom’s Taxonomy (recall, comprehension, application, analytical, synthesis, evaluation) Skills: Core departmental inpatient and outpatient procedures Skills Scale: Dreyfus Model of Skill Acquisition Overall self-assessment: ACGME general competencies

Duttweiler Locus of Control Index (1984) Previously validated instrument of LOC 28 question survey instrument Complete questions using scaled responses high internal LOC ~ score of 140. low internal LOC ~ score of 28.

Explanation of Locus Scoring Internal control index consists of 28 items Response alternatives fall along a five (5) point scale from “rarely” to “usually” The items are worded so that highly internally oriented subjects are expected to answer half at the “usually” end of the scale and the other half at the “rarely” end of the scale A maximum high internal response pattern would result in a score of 140. A minimum low internal response pattern would result in a score of 28.

Locus of Control results: Cohort 1, Year 1 and Year 2 –Relatively high amount of internal LOC –Low degree of variability in class Cohort 1, Year 1Cohort 1, Year 2 Mean Median Mode99 Std. Deviation

Locus of control results: Cohort 1, Year 1 Minimum score of 28 Maximum score of 140 Mean: Median: 106 Mode: 99 Std. Deviation: 8.68 Mean: Median: 106 Mode: 99 Std. Deviation: 8.68

LOC Results: Cohort 1, Year 2 Maximum score of 140 Minimum score of 28 Mean: Median: 104 Mode: 99 Std. Deviation: 8.54 Mean: Median: 104 Mode: 99 Std. Deviation: 8.54

Self Assessment Results: Cohort 1, Year 1 and 2 Low scores - participants have not encountered training or knowledge in these areas to this point Scoring: –0 – no prior exposure –1- rotation in medical –2- Rotation in residency –3- Rotation in both medical school and residency

Medical school/residency exposure/training results Top three areas with highest reported training/exposure Cohort 1, Year 1Cohort 1, Year 2 Maternity Care (mean=1.79, SD=0.94) Inpatient Pediatrics (mean=2.39, SD=0.76) Inpatient Pediatrics (mean=1.72, SD=1.0) Emergency Care (mean=2.35, SD=0.71) Surgical Care (mean=1.72, SD=1.0) Maternity Care (mean=2.32, SD=0.83)

Medical school/residency exposure/training results Top three areas with least training/exposure Cohort 1, Year 1Cohort 1, Year 2 Management of health systems (mean=0.17, SD=0.60) Oral health (mean=0.35, SD=0.75) Oral health (mean=0.38, SD=0.78) Endoscopy (mean=0.52, SD=0.57) Hematology (mean=0.41, SD=0.56) Infectious diseases (mean=0.52, SD=0.68)

Self Assessment: Current medical knowledge Scoring: 7 point scale –0 – little or nothing –1- “recall some basic facts” –2 - “can understand pathophysiology” –3 - “used knowledge in the care of some of my patients” –4 - “can recognize the signs and symptoms consistently” –5 - “can generalize from one patient to another” –6 - “can compare to best practices”

Results: knowledge areas 3 areas of highest reported knowledge/training (calculated by mean) Cohort 1, Year 1Cohort 1, Year 2 Maternity care (mean= 4.03, SD=1.05) Maternity care (mean=4.83, SD=1.00) Outpatient pediatrics (mean=3.93, SD=1.19) Outpatient pediatrics (mean=4.72, SD=0.96) Cardiology (mean=3.79, SD=0.91) Emergency Care (mean=4.66, SD=.86)

Results: knowledge areas 3 areas of lowest reported knowledge/training (calculated by mean) Cohort 1, Year 1Cohort 1, Year 2 Management of health systems (mean=1.72, SD=1.56) Oral health (mean=2.93, SD=1.58) Oral health (mean=2.14, SD=1.46) Opthamology (mean=3.17, SD=1.42) Hematology (mean=2.52, SD=1.15) Management of health systems (mean=3.21, SD=1.54)

Skill Assessment: outpatient/office- based procedures Scoring: Participants reported their skills in common outpatient procedures –0- “novice” –1- “advanced beginner” –2- “competent” –3 -“proficient” –4- “expert”

Results: outpatient/office based procedures –3 areas of highest reported skill level (calculated by mean) Cohort 1, Year 1Cohort 1, Year 2 Pap smear (mean=2.45, SD=0.95) Pap smear (mean=3.41, SD=0.68) Incision and drainage of abscess (mean=2.17, SD=0.89) Incision and drainage of abscess (mean=2.97, SD=0.82) EKG interpretation (mean=1.69, SD=1.17) Low-risk obstetrics (mean=2.76, SD=0.83)

Results: outpatient/office based procedures –3 areas of lowest reported skills (calculated by mean) Cohort 1, Year 1Cohort 1, Year 2 Vasectomy (mean=0.20, SD=0.41) Vasectomy (mean=0.38, SD=0.62) Thrombosed hemorrhoid incision (mean=0.41, SD=0.80) Aspiration of breast cyst (mean=0.66, SD=0.86) Reduction of uncomplicated dislocations (mean=0.52, SD=0.69) Thrombosed hemorrhoid incision (mean=0.72, SD=0.88)

Skill Assessment: Inpatient procedures Scoring: Participants reported their skills in common inpatient procedures –A response of “novice” was scored as “0”. –A response of “advanced beginner” was scored as “1”. –A response of “competent” was scored as “2”. –A response of “proficient” was scored as “3”. –A response of “expert” was scored as “4”.

Results: inpatient procedures –3 areas of highest reported skill level (calculated by mean) Cohort 1, Year 1Cohort 1, Year 2 Venipuncture (mean=1.86, SD=1.30) Vertex Delivery (mean=3.0, SD=0.71) Fetal Monitoring (mean=1.83, SD=1.00) Low Risk Obstetrics (mean=2.72, SD=0.80) Vertex Delivery (mean=1.72, SD=1.07) Fetal Monitoring (mean=2.69, SD=0.71)

Results: inpatient procedures –3 areas of lowest reported skill level (calculated by mean) Cohort 1, Year 1Cohort 1, Year 2 Forceps assisted vaginal delivery (mean=0.38, SD=0.86) Forceps assisted vaginal delivery (mean=0.59, SD=0.98) Central venous lines (mean=0.55, SD=0.74) Central venous lines (mean=0.86, SD=0.83) Chest tube placement (mean=0.59, SD=0.63) Chest tube placement (mean=0.90, SD=0.77)

Overall assessment of competency level Scoring: Participants reported their skill level –A response of “novice” was scored as “0”. –A response of “advanced beginner” was scored as “1”. –A response of “competent” was scored as “2”. –A response of “proficient” was scored as “3”. –A response of “expert” was scored as “4”.

Results: Overall ACGME Competency Assessment Top two highest and bottom two lowest reported scores: Highest: Lowest: Cohort 1, Year 1Cohort 1, Year 2 Communicating with patients and families (mean=2.48, SD=0.78) Communicating with patients and families (mean=3.00, SD=0.71) Engage other professionals (mean=2.24, SD=0.64) Engage other professionals (mean=2.76, SD=0.69) Cohort 1, Year 1Cohort 1, Year 2 Practicing cost effective health care (mean=1.59, SD=0.63) Facilitating learning of students/other professionals (mean=2.31, SD=0.76) Practicing continuous quality improvement (mean=1.72, SD=0.80) Practicing cost effective health care (mean=2.31, SD=0.89)

Next Steps Now-July 1, 2011: Cohort 2 participants will at year end retake the self assessment and locus instruments We will collect USMLE/AOA Board Scores and in-training scores July 1, 2011: Assessment of new incoming class of approximately 45 residents begins

Questions?

Theoretical References Bloom B. S. (1956). Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. New York: David McKay Co Inc. Dreyfus, Stuart E. and Dreyfus, Hubert L. (1980). A Five Stage Model of the Mental Activities Involved in Directed Skill Acquisition. Unpublished Report. University of California-Berkeley/US Air Force. Duttweiler, Patricia C. (1984). The Internal Control Index: a Newly Developed Measure of Locus of Control. Educational and Psychological Measurement. 44(2), p Trope, Y. (1982). Task Construction and Persistence as Means for Self-Assessment of Abilities. Journal of Personality and Social Psychology. 42(4), p