1. Effect of a PDA-assisted evidence based medicine course on knowledge of common clinical problems Roland Grad MDCM MSc FCFP FAAFP Yuejing Meng MSc Gillian Bartlett PhD Martin Dawes MD Pierre Pluye MD PhD Miriam Boillat MD Goutham Rao MD Roger Thomas MD PhD In Press: Family Medicine 2005;37(10):734-40.

2. What is the effect of a PDA- assisted EBM course in residency on objectively measured knowledge?

3. “Putting Evidence into Practice”  Controlled before/after  Four sites  44 residents  4 FP faculty

4. Methods 1.Knowledge test development by committee Hypertension – 18 questions Diabetes – 12 questions Risk assessment – 5 questions 2.Knowledge test validation FP faculty versus R1 versus Med 3 3.Before – after assessment of the effect of a PDA-assisted EBM course on knowledge  Two post tests, without and then with access to InfoRetriever

5. Intervention  Software training (2 x 1.5 hrs)  Integrated within an EBM course within a two-month FM rotation  Information hits tracked in log files

6. Timing of Outcome Measures Timing of Outcome Measures 2003 Jul A 1 2003 Aug 2003 Sep K 1 2003 Oct B 2003 Nov B 2003 Dec B 2004 Jan B 2004 Feb B 2004 Mar B 2004 Apr K 2 K 3 K = Knowledge (this presentation) A = Attitudes B = Behaviour (impact of information hits ) See Grad RM, Pluye P et al. Assessing the impact of clinical information-retrieval technology in a Family Practice Residency. Journal of Evaluation in Clinical Practice 2005. Available online.

7. Results

8. 44 eligible 1 st year residents July 2003 Five declined Two did not meet inclusion criteria 37 consenting residents Training Step 1 PDA, Lexidrugs, Praxis procedure tracking Training Step 2 How to Use InfoRetriever on PDA N = 37 R1 Intervention 1 EBM Course Sept-Nov 2003 N = 10 Intervention 2 EBM Course Feb-Apr 2004 N = 10 Progress of Residents Through The Trial Control group No EBM Course N = 17

9. Did residents use InfoRetriever on PDA?

10. Validation: Knowledge test scores of three groups at baseline Family Medicine Faculty (n=45) First Year Residents (n=37) Medical Students (n=24) Mean Score (95% CI) 20.5 (19.6-21.5) 17.1 [1][1] (16.2-18.0) 15.7 [2][2] (14.7-16.7) Percent Correct 64%54%49 % [1][1] p<0.0001 for observed difference in faculty compared to first year resident mean scores [2][2] p=0.033 for observed difference in first year resident compared to medical student mean scores

11. Test scores: Intervention v. control TestInterventionControl Baseline Test Score (95% CI) % Correct 18.2 (17.0-19.3) 56.9% n=20 15.9 (14.7-17.2) 49.7% n=17 First Post Test Score (95% CI) % Correct 20.5 (19.1-21.9) 64.1% n=17 18.8 (17.1-20.5) 58.8% n=16 Second Post Test Score (95 CI) % Correct 22.8 (21.3-24.3) 1 71.3% n=12 [1] p=0.046 for difference between second and first post test scores

12. Correct disease probability estimates among 12 residents with and without PDA access to clinical predictions rules for strep throat and CHD risk TestNumber of correct estimates (%) First post test (no access)21/60 (35%) Second post test with access to Clinical Prediction Rules 42/60 (70%)[1][1] [1] p=0.007 for observed difference in proportion of correct estimates

13. Conclusions  There was no effect of a PDA-assisted EBM course on knowledge test scores  Using the PDA during the test results in higher scores  Residents’ ability to estimate disease probability improved when they used PDA clinical prediction rules  This last conclusion is based on subgroup analysis, and requires further study

14. Funded by the Canadian Institutes of Health Research Thanks to the many residents & faculty who contributed their time and ideas: Armand Aalamian Randi Morris Suzanne Bouchard Maureen Doyle Sara Meltzer David Topps Gene Bailey

15. Clinical decision support system

16. Clinical information retrieval technology