Assessing verbal communication skills of medical students J Voges E Jordaan * L Koen DJH Niehaus Department of Psychiatry, University of Stellenbosch and Stikland Hospital * Biostatistics Unit: Medical Research Council, Bellville

Positioning of the study Large project:  Correlation of communication skills with academic performance of medical students Sub-studies:  Facial affect recognition  Oral examination marks in psychiatry  Non-verbal communication skills  Verbal communication skills

Introduction Communication is one of six required competencies identified by the ACGME Effective communication associated with:  Improved patient and doctor satisfaction  Treatment compliance  Strong predictor of medical school success Assessment of communication skills is complex and difficult to implement

Aim First phase: To evaluate the usability of the Liverpool Communication Skills Assessment Scale for assessing the communication skills of medical students of the University of Stellenbosch Second phase: To determine effectiveness of undergraduate medical students’ communication skills using the Liverpool Communication Skills Assessment Scale To determine if there is a correlation between communication skills and overall academic performance

Measurement equivalence Central issue in determining the applicability of instrument cross-nationally and cross-culturally Factors to consider:  Content equivalence  Semantic equivalence  Technical equivalence  Criterion equivalence  Conceptual equivalence (Flaherty et al, 1988)

Methods Subjects:  Medical students completing late rotation  5 min. semi-structured interview with patient that was videotaped  Permission granted by Faculty of Health Sciences and Ethics committee of SU Venue:  5-week Psychiatry rotation at Stikland hospital

Methods Assessment tool:  Liverpool Communication Skills Assessment Scale (LCSAS)  Consists of 12-items measuring several aspects of communication  Mixed method of using both a checklist and a rating approach  4-point ordinal rating scale ranging from Unacceptable to Good  Ease of use, acceptable reliability Raters:  2 independent raters, additional training  Third rater included, instruction given  Help sheet with additional descriptors to guide scoring Primary statistical evaluation:  Inter-rater reliability Marginal homogeneity (Chi-square statistic, p-value<0.01 as significant) Agreement (Cohen’s weighted Kappa index for ordinal data)

Results: Distribution of score by rater Intra-class correlation coefficient = 0.8 ( )

Results: Distribution of score by item

Agreement between raters Agreement for items that had marginal homogeneteity (Cohen’s weighted Kappa index for ordinal data, 95% confidence interval) ItemsRater 1 vs. Rater 2Rater 1 vs. Rater 3Rater 2 vs. Rater3 Item 1: Greeting0.71 ( )0.54 ( )0.57 ( ) Item 2: Introduction 0.76 ( )0.79 ( )0.92 ( ) Item 4: Eye-contact0.66 ( )0.38 ( )0.69 ( ) Item 8: Questions0.62 ( )0.37 ( )0.55 ( )

Discussion LCSAS evaluated for usability to measure communication skills in medical students Additional training Inclusion of additional rater and help sheet Total score: high level of correlation Inter-rater reliability  Marginal homogeneity 4 of 12 items  Agreement  Additional training – greater agreement Reliability of measure:  Continue with development and standardisation of assessment scale for use in South Africa  Training

Limitations and future directions Limitations:  Small sample  Inter-rater reliability Marginal homogeneity and agreement Training Future directions:  Re-evaluation of the scoring categories to promote understanding  Language  Gender  Culture

Conclusion Use of LCSAS in South African setting Correlation for total score Training necessary to improve agreement for each item Further development necessary Use in education context of South Africa by various health professionals

 Epstein, R.M. Campbell, T.L., Cohen-Cole, S.A., McWhinney, I.R. & Smilkstein, G. (1993). Perspectives on patient-doctor communication. Journal of Family Practice 37(4): 377–388.  Flaherty, J.A., Gaviria, F.M., Pathak, D., et al. (1988). Developing instruments for cross-cultural psychiatry needs. Journal of Nervous and Mental Disorders 176(5):  Humphris, G.M. & Kaney, S. (2001). The Liverpool Brief Assessment System for Communication Skills in the Making of Doctors. Advances in Health Sciences Education 6: 69–80.  Parker, G. (1993). On our selection: predictors of medical school success. Medical Journal of Australia 158(11): 747–751. Project supported by funding from FINLO Faculty of Health Sciences Selected references