1. User Satisfaction: Illusion or Paradox? Charlene R. Weir, PhD Professor Department of Biomedical Informatics 3515B

2. OVERVIEW Current state of the literature (2015 only)  Instruments used  Correlation with adoption Purpose and Related Theories  Proxy for adoption  Proxy for effectiveness Alternative View  Focus on Behavior (usage)  Motivation  Task-based assessments of effectiveness

3. EXAMPLE (VA)

4. Recent Studies(2015) Assessing User Satisfaction # of studies 32 Intention to use 12 Adoption Behavior (only usage) 7 Information Performance 5 Theories 15

5. What is the question that User Satisfaction is supposed to answer? A proxy for adoption? A proxy for effectiveness?

6. A Proxy for Adoption What IS Adoption?  Intention to Use  Usage/Behavior  Customization What are relevant theories?  Technology Adoption Model  Universal Theory of ***

7. DIFFUSION OF INNOVATION A staged communication process Early Innovators

8. Innovation Characteristics Relative Advantage – compared to other things Compatibility - How consistent the innovation is with the values, experiences of adopters. Complexity - How difficult the innovation is to understand Triability - The extent to which the innovation can be tested or experimented with Observability - The extent to which the innovation provides tangible results.

9. MOTIVATIONAL THEORIES

10. Theory of Planned Behavior Behavioral belief / Perceptions of Outcomes: beliefs regarding outcomes of XXX PLUS the subjective probability that the behavior will produce that outcome. (CPOE decreases errors, improves communication, improves quality of care) Attitude toward behavior: Perceived value for the outcome (good thing/bad thing) Normative beliefs: perception of social influence Perceived behavioral control: beliefs about it actually working (e.g. support, mandated, resources). Control beliefs: The concept of perceived behavioral control is conceptually related to self-efficacy.

11. UTAUT ( Unified Theory of Acceptance and Use of Technology ) Venkatesh, V., Morris, M.G., Davis, F.D., and Davis, G.B.(2003) “User Acceptance of Information Technology: Toward a Unified View,” MIS Quarterly MODERATORS

12. TAM Technology Acceptance Model

13. SUS (System Usability Scale) I think that I would like to use this system frequently I found the system unnecessarily complex I thought the system was easy to use I think that I would need the support of a technical person to be able to use this system I found the various functions were integrated I thought there was too much inconsistency in the system I would imagine that most people would learn to use this system very quickly I found the system very cumbersome to use I felt very confident using the system I needed to learn a lot of things before I could get going

14. What is WRONG with this picture? Why do we care? Most usage is mandated in clinical information systems Adoption measures are not correlated with outcomes Adoption is not correlated to implementation strategies Models developed to PREDICT usage do not predict “making decisions better,” “workflow more efficient” or “enhanced coordination” Can’t be used for design Results are always in the 3.2 to 4.2 range (1 to 5 scale) We already KNOW the theories are empirically validated

15. A Proxy for Effectiveness What is effectiveness?  System performance  IT support  Information Management tasks  Documentation Goals  Communication/Coordination What are relative theories?  DeLone and McLeon  Task/Technology Fit  Information Theory

16. DeLone W, McLean E. (2003) The DeLone and McLean Model of Information Systems Success: A Ten-Year Update

17. Task/Technology Fit Goodhue & Ammenworth

18. WHAT ARE the NET BENEFITS of Clinical IT?

19. Example: Assessing Satisfaction and User Behavior for CPRS Adoption

20. Methods – Task Instrument Interviews of 24 Primary Care Practitioners Week-long observation of 2 primary clinics Action Identification Interviews  “Describe what you are doing” Qualitative Analysis Generation of 34 items Factor Analysis with 360 providers Reduction to 22 items (TASK)

21. Theory of Action Identification Vallacher R, Wegner D. What do people think they are doing? Action identification and human behavior. Psych Rev. 1987;94:3–15 Action identities  “what are you doing?”  “ You are doing XX in order to do XX?”  “ You are doing XX BY DOING YY”? Hierarchical construction  Goals/Values on top  Actions at bottom Experts report high levels; novices low levels

22. Typical Action “Identities ” Reporting required information Making a decision about the best course of treatment Looking for patient data Checking over the patient status Becoming familiar with the patient Finding info to teach pt. Looking for the evidence on the best treatment Determining the # of procedures done per day Looking up local policies

23. Results

24. Methods – Validation N=124  26 Primary Care Practitioners  18 Other specialists  14 Pharmacists  44 RNs  22 Residents Pre/Post CPRS Implementation 92% response rate

25. Measures of User Satisfaction Task Effectiveness (TASK; n= 22 items)  “Information Tasks”  Three questions each on ratings of effectiveness Communication Documentation Decision-Making Workload Tracking Policy Compliance Data Review  Total Summed: alpha = 0.90 Usability (3 items)  Easy to use, minimal effort; easy to navigate  Total Summed: alpha = 0.93

26. Measures of User Satisfaction Intrinsic Motivation (AFFECT)  Three items assessing affective experience Interest Enjoyment Fun  Items summed to form scale: alpha = 0.68 User Satisfaction (GENERAL)  9 items (Doll, et al) (comprehensive, accurate, easy to use, useful, improves my work)  1 factor (alpha= 0.84)

27. Correlation Results

28. RESULTS: User Satisfaction Measures

29. Early vs Late Adopters: Progress Notes in Primary Care

30. Level of Adoption Personal Customization Sum of 12 customization options used

31. RESISTANCE RATINGS Taken from Reactance Theory  Reactance results from a threat to control  Results in demeaning the messenger; dismissing the message, avoiding the threat, and reasserting control (refusals, hording paper) Four members of the implementation team rated 24 primary care providers on: 1 (none) to 7 (very high) scale Ratings were averaged.

32. RESISTANCE RATINGS Correlation Results

33. STAGES OF CHANGE Prochaska & DiClemente Combines many motivational theories Proposes four stages in change process, characterized by attitude, intentions and behavior.  Pre-Contemplation - not concerned, little attention  Contemplation - thinking about the pros & cons  Action - formed an intention; making plans  Maintenance - behavior in place; well-learned

34. STAGES OF CHANGE They checked one (out of 5) I use the system only when I have to and I do not intend to try to expand my use in any way. I see no reason to learn much more about it. I have begun trying to integrate using the computer in my daily work processes ad am now partially successful. I am learning more about it all the time. Even if the system were not mandated, I would use it. My work is extensively dependent on the computer.

35. Stage of Change Mean values across stage for each measure (ANOVA) A good measure “picks up” the change across stages

36. Involvement/Volunteer Asked if they wanted to get more information and would like to volunteer. Involvement Group = either question (n=57) Non-involvement group (n=20) = checked none

37. INVOLVEMENT

38. So What? These results had NO impact on developers Did not inform implementation Might be used to compare across settings Need to develop metrics that can relate Usability measures to User Satisfaction ANSWER:  Focus on Tasks – relate them to goals  Link Usability Assessments to these Tasks  Link tasks to implementation strategies  Link tasks to clinical outcomes

39. Questions?