Cognitive Informatics for Biomedicine – Chapter 5 Evaluation of Health Information Technology: Methods, Frameworks and Challenges By – Utkrisht Rajkumar Cognitive Informatics for Biomedicine – Chapter 5

Cognitive Informatics for Biomedicine – Chapter 5 Background An increase in Healthcare Information Technology spurred by American Reinvestment and Recovery Act Health information technology (HIT) implementation  mixed results Two main evaluation methods: Evaluation of systems: usability and human computer interaction Generic usability: comprehensive perspective of HIT systems in clinics Cognitive Informatics for Biomedicine – Chapter 5

HIT Evaluation components: How we study effects of HIT on structure, processes, or outcomes? System functionality Impact of user interface on work activities Discovering specific interface and system issues that affect work Cognitive Informatics for Biomedicine – Chapter 5

Classification of evaluation methods Cognitive Information for Biomedicine – pg. 84 Cognitive Informatics for Biomedicine – Chapter 5

Analytical Approaches Using usability experts, domain experts, software designers, etc. Three types: Task-analytic Inspection-based Model-based Cognitive Informatics for Biomedicine – Chapter 5

Analytical Approaches Task Analytics Evaluate existing techniques to better understand rational behind: People’s goals of performing a task Motivations behind their goals How they perform these tasks Cognitive Informatics for Biomedicine – Chapter 5

Analytical Approaches Task Analytics Hierarchical Task Analysis: Break into sub-tasks. Designed to identify specific training needs Used extensively in the design and evaluation of interfaces Cognitive Task Analysis Extension of general task analysis Used to develop a comprehensive understanding of thought process Cognitive Informatics for Biomedicine – Chapter 5

Classification of evaluation methods Cognitive Information for Biomedicine – pg. 84 Cognitive Informatics for Biomedicine – Chapter 5

Analytical  Inspection based analysis Involves one or more experts playing the role of a user Used to identify potential usability and interaction problems Alternative to recruiting potential users to test the usability of a system. Cognitive Informatics for Biomedicine – Chapter 5

Analytical  Inspection based analysis Heuristic Evaluation: Utilize a small set of experts to evaluate a user interface Based on their understanding of a set of heuristic principles Cognitive Walkthrough: Analyzing the behaviors of users as they complete a scenario of use Focused on identifying the usability and comprehensibility of a system Cognitive Informatics for Biomedicine – Chapter 5

Classification of evaluation methods Cognitive Information for Biomedicine – pg. 84 Cognitive Informatics for Biomedicine – Chapter 5

Analytical  Model based Uses predictive modeling to measure efficiency of interfaces. Used for evaluating routine, expert task performance. Ex: How can the keys of a medical device interface by optimally organized? Used to compare the data entry efficiency between different interfaces Cognitive Informatics for Biomedicine – Chapter 5

Analytical  Model based Goals, Operators, Methods and Selection Rules (GOMS) predict the time taken to complete a task by a skilled/expert user Execution of tasks can be represented as a sequence of cognitive operations Fitts Law Predict human motor behavior Predict time taken to acquire target Ex: Develop predictive model of time to acquire a target using a mouse Cognitive Informatics for Biomedicine – Chapter 5

Classification of evaluation methods Cognitive Information for Biomedicine – pg. 84 Cognitive Informatics for Biomedicine – Chapter 5

Cognitive Informatics for Biomedicine – Chapter 5 Usability Testing Used for evaluating the usability of HIT systems Two types: Field/Observational studies General approaches Cognitive Informatics for Biomedicine – Chapter 5

Usability  Field Observational Shadowing A researcher following a participant over an extended period of time Focus on collecting data about a single participant Time and Motion Studies Deeper understanding of the impact of clinical work activities; Ex: changes on clinical efficiency due to the implementation of a new EHR system Cognitive Informatics for Biomedicine – Chapter 5

Usability  General Approaches Interviews Elicit information about opinions and perspectives of participants Opinion of the clinical workflow and opportunities for improvement Verbal think aloud Data on the thought processes that underlie human actions Surveys Widespread use to ease of administration and limited time required to complete Cognitive Informatics for Biomedicine – Chapter 5

Considerations for HIT Evaluation Integrated approach of multiple evaluation studies Preliminary considerations for evaluations: Environment of evaluation study Use of a framework to guide the evaluation process Challenges for conducting HIT evaluation studies Cognitive Informatics for Biomedicine – Chapter 5