The Canadian KTpush project 41 FPs in practice 1-year mixed methods study PUSH and PULL components PUSH: 4,548 rated alerts PULL: 1,767 rated searches / 3,300 rated information hits

PUSH PUSH Context

NMeanRange No. rated InfoPOEMs® / MD41 MD111 (51.5) No. Days / MD41 MD314.2 (54.4) PUSH Context

Relevance vs. positive cognitive impact * Excluding 1,018 InfoPOEM® ratings of ‘No Impact’ PUSH Context Cognitive impact * n= 3,530 Relevant (partially or totally) n= 2,902 (%) Not relevant n= 628 (%) My practice was (will be) changed and improved 866 (29.8)56 (8.9) I learned something new2,076 (71.5)467 (74.4) This information confirmed I did (will do) the right thing 1,292 (44.5)127 (20.2) I was reassured1,478 (50.9)159 (25.3) I am reminded of something I already knew 844 (29.1)98 (15.6) I am motivated to learn more1,393 (48.0)177 (28.2)

Relevance vs. negative cognitive impact Cognitive impact n= 3,530 Relevant (partially or totally) n= 2,902 (%) Not relevant n= 628 (%) I am dissatisfied, as this information has no impact on my practice 53 (1.8)73 (11.6) I am dissatisfied, as there is a problem with this information 151 (5.2)107 (17.0) I disagree with this information25 (0.9)12 (1.9) I think this information is potentially harmful45 (1.6)20 (3.2) PUSH Context

Relevance vs. use of InfoPOEMs® How the information will be used Relevant for at least one of your patients? # Ratings (%)For thinking To justify or maintain To modify Totally relevant1,671 (36.7)1,275 (76.3)992 (59.4)655 (39.2) Partially relevant1,456 (32.0)1,141(78.4)597 (41.0)410 (28.2) Not relevant1,421 (31.3)0 (N/A) Total4,5482,4161,5891,065 Ratings (%): column percentage Thinking (%), Justify or maintain (%), Modify (%): row percentage PUSH Context

Positive cognitive impact vs. use of InfoPOEMs® for a specific patient Excluding 1,018 InfoPOEM® ratings of ‘No Impact’ PUSH Context Cognitive impact n= 3,530 For thinking about this patient n= 2,285 (%) To justify or maintain management n= 1,521 (%) To modify management n= 1,051 (%) My practice was (will be) changed and improved 694 (30.4)364 (23.9)671 (63.8) I learned something new1,689 (73.9)899 (59.1)932 (88.7) This information confirmed I did (will do) the right thing 1,015 (44.4)1098 (72.2)331 (31.5) I was reassured1,214 (53.1)1082 (71.1)530 (50.4) I am reminded of something I already knew 671 (29.4)636 (41.8)233 (22.2) I am motivated to learn more1,176 (51.5)681 (44.8)690 (65.7)

Situational relevance CognitionApplication PULL

2,131 Searches 1,767 Rated searches 3,300 Rated hits 329 Deleted searches 953 Deleted hits 35 Unrated searches 76 Unrated hits 2 Excluded searches (bug) 6 Excluded hits (bug) 37 Excluded searches (logic error) 161 Excluded hits (logic error) Untrackable Response rate 83% PULL Context

Searches and hits per MD NMeanRange No. rated searches / MD40 MD44.2 (30.8)6-148 No. rated hits / MD40 MD82.5 (63.7)9-294 No. rated hits / rated search 1,767 rated searches 3,300 rated hits 1.9 (1.7)1- 22 PULL Context

AcquisitionCognitionApplication PULL Context

Task Analysis: Reasons for Searches in Family / General Practice Address a clinical question/problem/ decision making about a specific patient1,310 (74%) Look up something I had forgotten672 (38%) Share information with a patient/caregiver624 (35%) Exchange information with other health professionals520 (29%) Search in general or for curiosity496 (28%) Fulfill an educational or research objective434 (25%) Plan, manage, coordinate, delegate or monitor tasks with other health professionals 197 (11%) AcquisitionCognitionApplication PULL Context

Number of reasons given for searches PULL Context

AcquisitionCognitionApplication Types of Cognitive Impact 7,275 reported cognitive impacts linked to 3,300 rated hits This information confirmed I did (will do) the right thing151646% I was reassured146845% I learned something new124638% I recalled something113634% My practice was (will be) changed and improved96329% No Impact78024% Negative Impact (a composite of four types)1665% PULL Context

AcquisitionCognitionApplication 166 of 3,300 rated hits (5%) contain reports of Negative Impact I was dissatisfied, as this information had no impact on my practice79 (2.4%) I was dissatisfied, as there was a problem with this information67 (2.0%) I think this information is potentially harmful13 (0.4%) I disagree with this information7 (0.2%) PULL Context

Too much information10.0% Not enough information310.9% Information poorly written20.1% Information too technical10.0% Other problem (please specify)411.2% I was dissatisfied, as there was a problem with this information PULL Context

AcquisitionCognitionApplication 1,708 rated hits (52%) were (or will be) used for a specific patient PULL Context

How does achieving the search objective influence cognitive impact? 3,300 hits in 1,767 searches Objective met: n= 2,482 hits (75.2%) in 1,336 searches (75.6%) PULL Context Cognitive impact n=3,300 Objective met n= 2,482 (%) Objective not met n= 818 (%) My practice was (will be) changed and improved899 (36.2)64 (7.8) I learned something new1,104 (44.5)142 (17.4) This information confirmed I did (will do) the right thing1,378 (55.5)138 (4.2) I was reassured1,351 (54.4)117 (14.3) I recalled something1,021 (41.1)115 (14.1) I am dissatisfied, as this information has no impact on my practice 10 (0.4)69 (8.4) I am dissatisfied, as there is a problem with this information21 (0.9)46 (5.6) I disagree with this information6 (0.2)1 (0.1) I think this information is potentially harmful11 (0.4)2 (0.2) This item of information had no impact at all on me or my practice 288 (11.6)492 (60.2)

How is Cognitive impact associated with the use of information for a specific patient? PULL Context Cognitive impact n=3,300 Used for a specific patient n= 1,708 (51.8%) Not used for a specific patient n=1,592 (48.2%) My practice was (will be) changed and improved709 (41.5)254 (16.0) I learned something new756 (44.3)490 (30.8) This information confirmed I did (will do) the right thing 1081 (63.3)435 (27.3) I was reassured1027 (60.1)441 (27.7) I recalled something803 (47.0)333 (20.9) I am dissatisfied, as this information has no impact on my practice 23 (1.4)56 (3.5) I am dissatisfied, as there is a problem with this information 22 (1.3)45 (2.8) I disagree with this information2 (0.1)5 (0.3) I think this information is potentially harmful5 (0.3)8 (0.5) This item of information had no impact at all on me or my practice 106 (6.2)674 (42.3)

No cognitive impact vs. use of information Did you (will you) use this information for a specific patient? YES (N=1,708)NO (N=1,592) This information had no impact at all on me or my practice (N=780) PULL Context

From PUSH to PULL

DYADS n=21 PUSH 4,937 alerts ( InfoPOEMs® ) read PULL 1,205 retrieved InfoPOEMs®

PurposefulSerendipitous Critical thinking dyads n=4 Known-item dyads n=6 Recalled when re-read n=3 Not recalled at all n=7 DYADS n=21* *One participant did not clearly remember one dyad.

Average time interval between date of reading an InfoPOEM® and the memory test Read and Rated (n=240) Read Not Rated (n=240) Average number of days between reading and memory test Minimum number of days6014 Maximum number of days172164

Ability of 24 family doctors to remember whether an InfoPOEM® was previously read (or never read) Three Categories of Delivered InfoPOEMs® Read and RatedRead Not RatedNever Read Number correctly remembered / total 166 / 240 (69%)181 / 240 (75%)165 / 240 (69%) Range of correct answers per MD 3-10 / / / 10

Effect of recency of reading on the probability of obtaining an incorrect answer to the question “Have you read this InfoPOEM®?” Logistic Regression Model Model Incorrect Answer = b0+ b1* Rated + b2* Time interval Parameter estimate (SD)P Value Odds ratio [95% C.I.] Rated (1) vs. Read Not Rated (0) (0.32) [0.41,1.42] Time interval0.008 (0.003) [1.00,1.02]

Comparing PUSH and PULL

My practice was (will be) changed and improved Push n=922 Pull n=963 Changed and improved n (%) Commitment n (%) Changed and improved n (%) Commitment n (%) Diagnostic approach75 (8.1)90 (9.8)189 (19.6)114 (11.8) Therapeutic approach311 (33.7)357 (38.7)422 (43.8)290 (30.1) Health education / Disease prevention 184 (20.0)178 (19.3)165 (17.1)121 (12.6) Prognostic approach100 (10.9)107 (11.6)87 (9.0)64 (6.7) Other (please specify)16 (1.7)51 (5.5)20 (2.1)78 (8.1) % are row percentages

Dissatisfaction (problem with this information) Type of dissatisfaction n=258 Push Yes n (%) Pull Yes n (%) Too much Information2 (0.8)1 (0.0) Not enough information140 (54.3)31 (0.9) Information poorly written33 (12.8)2 (0.1) Information too technical2 (0.8)1 (0) Other problem (please specify)107 (41.5)41 (1.2)

PUSH n=4,500 rated s PULL n=3,300 rated hits DYADS n=21

Do physicians manually update downloadable software on PDA? About 25% of FPs never or rarely updated on their own Many software updates were never installed Only a small number of FPs made all requested updates on their own

Semi-automated updating

Manual updating

When are searches rated?