1. Cognitive Informatics for Biomedicine – Chapter 5
Evaluation of Health Information Technology: Methods, Frameworks and Challenges
By – Utkrisht Rajkumar
Cognitive Informatics for Biomedicine – Chapter 5

2. Cognitive Informatics for Biomedicine – Chapter 5
Background
An increase in Healthcare Information Technology spurred by American Reinvestment and Recovery Act
Health information technology (HIT) implementation  mixed results
Two main evaluation methods:
Evaluation of systems: usability and human computer interaction
Generic usability: comprehensive perspective of HIT systems in clinics
Cognitive Informatics for Biomedicine – Chapter 5

3. HIT Evaluation components:
How we study effects of HIT on structure, processes, or outcomes?
System functionality
Impact of user interface on work activities
Discovering specific interface and system issues that affect work
Cognitive Informatics for Biomedicine – Chapter 5

4. Classification of evaluation methods
Cognitive Information for Biomedicine – pg. 84
Cognitive Informatics for Biomedicine – Chapter 5

5. Analytical Approaches
Using usability experts, domain experts, software designers, etc.
Three types:
Task-analytic
Inspection-based
Model-based
Cognitive Informatics for Biomedicine – Chapter 5

6. Analytical Approaches Task Analytics
Evaluate existing techniques to better understand rational behind:
People’s goals of performing a task
Motivations behind their goals
How they perform these tasks
Cognitive Informatics for Biomedicine – Chapter 5

7. Analytical Approaches Task Analytics
Hierarchical Task Analysis:
Break into sub-tasks.
Designed to identify specific training needs
Used extensively in the design and evaluation of interfaces
Cognitive Task Analysis
Extension of general task analysis
Used to develop a comprehensive understanding of thought process
Cognitive Informatics for Biomedicine – Chapter 5

8. Classification of evaluation methods
Cognitive Information for Biomedicine – pg. 84
Cognitive Informatics for Biomedicine – Chapter 5

9. Analytical  Inspection based analysis
Involves one or more experts playing the role of a user
Used to identify potential usability and interaction problems
Alternative to recruiting potential users to test the usability of a system.
Cognitive Informatics for Biomedicine – Chapter 5

10. Analytical  Inspection based analysis
Heuristic Evaluation:
Utilize a small set of experts to evaluate a user interface
Based on their understanding of a set of heuristic principles
Cognitive Walkthrough:
Analyzing the behaviors of users as they complete a scenario of use
Focused on identifying the usability and comprehensibility of a system
Cognitive Informatics for Biomedicine – Chapter 5

11. Classification of evaluation methods
Cognitive Information for Biomedicine – pg. 84
Cognitive Informatics for Biomedicine – Chapter 5

12. Analytical  Model based
Uses predictive modeling to measure efficiency of interfaces.
Used for evaluating routine, expert task performance.
Ex: How can the keys of a medical device interface by optimally organized?
Used to compare the data entry efficiency between different interfaces
Cognitive Informatics for Biomedicine – Chapter 5

13. Analytical  Model based
Goals, Operators, Methods and Selection Rules (GOMS)
predict the time taken to complete a task by a skilled/expert user
Execution of tasks can be represented as a sequence of cognitive operations
Fitts Law
Predict human motor behavior
Predict time taken to acquire target
Ex: Develop predictive model of time to acquire a target using a mouse
Cognitive Informatics for Biomedicine – Chapter 5

14. Classification of evaluation methods
Cognitive Information for Biomedicine – pg. 84
Cognitive Informatics for Biomedicine – Chapter 5

15. Cognitive Informatics for Biomedicine – Chapter 5
Usability Testing
Used for evaluating the usability of HIT systems
Two types:
Field/Observational studies
General approaches
Cognitive Informatics for Biomedicine – Chapter 5

16. Usability  Field Observational
Shadowing
A researcher following a participant over an extended period of time
Focus on collecting data about a single participant
Time and Motion Studies
Deeper understanding of the impact of clinical work activities;
Ex: changes on clinical efficiency due to the implementation of a new EHR system
Cognitive Informatics for Biomedicine – Chapter 5

17. Usability  General Approaches
Interviews
Elicit information about opinions and perspectives of participants
Opinion of the clinical workflow and opportunities for improvement
Verbal think aloud
Data on the thought processes that underlie human actions
Surveys
Widespread use to ease of administration and limited time required to complete
Cognitive Informatics for Biomedicine – Chapter 5

18. Considerations for HIT Evaluation
Integrated approach of multiple evaluation studies
Preliminary considerations for evaluations:
Environment of evaluation study
Use of a framework to guide the evaluation process
Challenges for conducting HIT evaluation studies
Cognitive Informatics for Biomedicine – Chapter 5