1. A feasibility study of an automated practice-based survey
Sharon Johnston, Sabrina Wong, Fred Burge, William Hogg
North American Primary Care Research Group Annual Conference
Montreal, QC
November 18, 2017
MEASURING AND IMPROVING THE PERFORMANCE OF PRIMARY HEALTH CARE IN CANADA

2. Challenge: sustainable patient experience data collection
Study objective: demonstrate the feasibility and acceptability of an automated patient survey system (adopted from an existing automated appointment reminder system)

3. Methods Setting: Three Canadian health regions in BC, ON, & NS
Recruitment: Patients completing paper-based waiting room survey at their practices (n=87 practices)
Automated survey
5 questions administered within 72 hours of patient visit
Offered in English/French and phone/ formats
Outcomes
Response rates: mode, age group, gender, & income
Concordance between paper and APS survey responses

4. Automated Survey Questions
How good was your family doctor or nurse at giving you enough time?
How good was your family doctor or nurse at explaining tests and treatments?
Does your family doctor or nurse tell you about side effects you might get from a medicine?
Were there times when the person you were seeing did not know about changes in your treatment plan that another person recommended?
Were there times when the person you were seeing did not have access to your recent tests or exam results?

5. APS Respondents & Mode Preference
Eligible to participate
1929
Number of who consented
871 (56%)
Mode preference:
606 (69.6%)
Phone
265 (30.4%)
Response rate
(55.6%)
Completion rate*
(% of those who responded to APS survey)
470 (97.1%)
Note: *surveys were complete if participants answered 5/5 questions
APS Respondents
Mode Preference
n (%)
n(%)
Phone
Eligible to participate
1929
Consented to participate
871
606 (69.6%)
265 (30.4%)
Response rate
484 (55.6%)
Completion rate*
(% of those who agreed to APS survey)
470 (97.1%)
369 (60.9%)
101 (38.1%)

6. APS Respondents & Mode Preference
Phone n (%)
n (%)
18-24 years
3 (10.3%)
26 (89.7%)
25-64 years
133 (24.1%)
420 (75.9%)
65-74 years
78 (42.2%)
107 (57.8%)
75+ years
45 (53.6%)
39 (46.4%)
Male
101 (38.5%)
161 (61.5%)
Female
154 (26.5%)
427 (73.5%)
Transgender --
Income < $20K
50 (44.2%)
63 (55.8%)
Income $20K - $40K
64 (38.8%)
101 (61.2%)
Income $40K - $60K
47 (29.0%)
115 (71.0%)
Income $60K - $100K
43 (20.9%)
163 (79.1%)
Income $100K +
23 (16.9%)
113 (83.1%)
0 or 1 chronic conditions
43 (17.8%)
199 (82.2%)
2 chronic conditions
42 (33.3%)
84 (66.7%)
3 or more chronic conditions
170 (35.8%)
305 (64.2%)
Respondents significantly preferred over phone for all age groups, except for those aged 75+

7. APS Completion Rates
Significantly better completion rates among responders for:
Age groups years and 75+ years
Sexes
For most income groups
For all chronic conditions
Completed APS
Phone
n (%)
Overall
101 (38.1%)
369 (60.9%)
18-24 years
0 (0%)
11 (42.3%)
25-64 years
50 (37.6%)
263 (62.6%)
65-74 years
35 (44.9%)
62 (57.9%)
75+ years
12 (26.7%)
25 (64.1%)
Male
28 (27.7%)
97 (60.2%)
Female
68 (44.2%)
262 (61.4%)
Transgender -
< $20K
23 (46%)
29 (46%)
$20K - $40K
23 (35.9%)
58 (57.4%)
$40K - $60K
18 (38.3%)
74 (64.3%)
$60K - $100K
16 (37.2%)
108 (66.3%)
$100K +
10 (43.5%)
73 (64.6%)
0-1 chronic conditions
12 (27.9%)
115 (57.8%)
2 chronic conditions
14 (33.3%)
48 (57.1%)
3+ chronic conditions
71 (41.8%)
197 (64.6%)

8. Agreement between paper and APS responses
Question %
Concordant
Weighted Kappa (quadratic)
Given enough time
75.8%
0.52
Explained tests and treatments
72.8%
0.45
Told about potential side effects from medications
72.5%
0.53
Times when provider didn’t have access to recent tests or exam results
57.1%
0.12
Times when provider did not know about changes in treatment plan that another person recommended
61.8%
0.16
For 3 questions, moderate concordance was seen between the waiting room paper survey and the APS
Two questions saw slight agreement

9. Directionality of agreement between paper and APS
Phone
More
-ve
% Concordant
More +ve
Given enough time
10.0%
64.6%
25.0%
12.0%
78.8%
9.2%
Explained tests and treatments NS
Told about potential side effects from medications
Times when provider didn’t have access to recent tests or exam results
Times when provider did not know about changes in treatment plan that another person recommended
44.2%
45.3%
10.5%
28.3%
66.1%
5.6%
Mode of phone yielded more positive responses for enough time
More negative responses for knowing about changes in treatment plan

10. Lessons learned Automated survey is feasible
Language preference easy to incorporate
response rates appear significantly better than phone
Age and income differences in vs phone preference as well as response rates
Potential mode effects

11. Future directions: Direct from EMR
Deploy survey directly from EMR like appointment reminder systems or from patient registries:
Deployed by preset parameters at specified times and intervals, e.g.:
All or random random selection within age group
If not seen in two years or missing recommended screening
After visit for PHE
APS linked to routine automated patient outreach:
Appointment reminders
Preventive care reminders