MEQ Analysis

Outline Validity Validity Reliability Reliability Difficulty Index Difficulty Index Power of Discrimination Power of Discrimination

Validity “evidence present to support or refute the meaning assigned to assessment results” face validity face validity content validity content validity criterion-related validity criterion-related validity construct validity construct validity

Face Validity high face validity high face validity  seemed valid (only)  ~ person knee jerk & nervous system knee jerk & nervous system  for doctor => high face validity  for lay people => low face validity MEQ => high face validity MEQ => high face validity

Content Validity ~ sample & population ~ sample & population MCQ => high content validity MCQ => high content validity MEQ => low content validity MEQ => low content validity

Validity “evidence present to support or refute the meaning assigned to assessment results” face validity face validity content validity content validity criterion-related validity criterion-related validity construct validity construct validity - don’t need the score - before using the test - score needed - after using the test

Criterion-Related Validity Predictive validity Predictive validity  MEQ score & close observation score of real practice at ER Concurrent Validity Concurrent Validity  MEQ score & VIVA score in the same topic Statistic = correlation coefficient Statistic = correlation coefficient

Construct Validity based on theory based on theory  communication skill ~ leadership correlation OSCE : communication VS questionnaire : leadership correlation OSCE : communication VS questionnaire : leadership  good ethics ~ beloved doctor MEQ : medical ethics VS questionnaire : beloved doctor? MEQ : medical ethics VS questionnaire : beloved doctor?

Reliability stability stability internal consistency internal consistency equivalent equivalent

Stability test-retest reliability test-retest reliability parallel form reliability parallel form reliability intra-rater reliability intra-rater reliability  ~ scoring key statistics : correlation coeff. statistics : correlation coeff.  0-1

Internal Consistency [homogeneity] [homogeneity] item - item correlation item - item correlation item - total correlation item - total correlation split half correlation split half correlation

Item-Item Correlation each item each item  Dichotomous Phi Correlation Phi Correlation  Interval Pearson’s Product Moment Correlation Pearson’s Product Moment Correlation whole test whole test  Mean of...

Item-Total Correlation each item each item  Dichotomous Point Biserial Correlation Point Biserial Correlation  Interval Pearson’s Product Moment Correlation Pearson’s Product Moment Correlation whole test whole test  Mean of...

Spilt Half Reliability Dichotomous Dichotomous  Kuder-Richardson 20 (KR 20) Interval Interval  Kuder-Richardson 21 (KR 21)  Cronbach’s alpha coefficient

Equivalent parallel item on alternate form reliability parallel item on alternate form reliability inter-rater reliability inter-rater reliability  agreement  kappa

Difficulty Index [p] [p] (mean H + mean L)/2(full score) (mean H + mean L)/2(full score) p = 1 => very easy p = 1 => very easy p = 0 => very difficult p = 0 => very difficult expecting p = expecting p =  must => 0.7  should => 0.5

Power of Discrimination [r] [r] (mean H - mean L)/(full score) (mean H - mean L)/(full score) > 0.40 => very good item > 0.40 => very good item => good item => good item => borderline => borderline poor item poor item

Conclusion Validity Validity Reliability Reliability Difficulty Index Difficulty Index Power of Discrimination Power of Discrimination