©2010 John Wiley and Sons Chapter 11 Research Methods in Human-Computer Interaction Chapter 11- Analyzing Qualitative Data

©2010 John Wiley and Sons Chapter 11 Overview Introduction Stages of qualitative analysis Grounded theory Content analysis Analyzing text content Analyzing multimedia content

©2010 John Wiley and Sons Chapter 11 Stages of qualitative analysis Identify components of the substance Study properties and dimensions of each component Understand and make inference about the substance

©2010 John Wiley and Sons Chapter 11 Grounded theory An inductive research method

©2010 John Wiley and Sons Chapter 11 Procedures of grounded theory open coding development of concepts grouping concepts into categories formation of a theory

©2010 John Wiley and Sons Chapter 11 Grounded theory Advantages – a systematic approach to analyzing qualitative, mostly text-based, data, – generating theory out of qualitative data that can be backed up by ample evidence of the coding – Interplay between data collection and analysis

©2010 John Wiley and Sons Chapter 11 Grounded theory Disadvantages – Researcher can be overwhelmed by the details of the data – The theory generated is hard to evaluate – Findings may be subject to bias

©2010 John Wiley and Sons Chapter 11 Content analysis A more specific view: a systematic, replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding A broader view: any technique for making inferences by objectively and systematically identifying specified characteristics of messages

©2010 John Wiley and Sons Chapter 11 Content categories

©2010 John Wiley and Sons Chapter 11 Preparing for content analysis Define the data set Define the population Clean up the data Understand the context of the data

©2010 John Wiley and Sons Chapter 11 Analyzing text data A priori coding – Identify coding categories – Coding – Reliability check Emergent coding – Multiple coders identify coding categories based on subset of data – Consolidate category list – Code a subset of data – Reliability check – Repeat the process until satisfactory result is met – Code the rest of the data

©2010 John Wiley and Sons Chapter 11 Identify coding categories Theoretical framework Researcher-denoted concepts In-vivo codes Building a code structure (nomenclature)

©2010 John Wiley and Sons Chapter 11 Coding the text Look for key items

©2010 John Wiley and Sons Chapter 11 Coding the text Ask questions about the data Making comparisons – Between different coding category – Between different participant group – Between existing data and previous literature Using computer software

©2010 John Wiley and Sons Chapter 11 Ensure high quality analysis Validity – constructing a multi-faceted argument in favor of your interpretation of the data – Constructing a database – Data source triangulation – Interpretation should account for as much as possible of the data – Alternative interpretations may also help

©2010 John Wiley and Sons Chapter 11 Ensure high quality analysis Reliability check – Stability also called intra-coder reliability examines whether the same coder rates the data in the same way throughout the coding process – Reproducibility also called inter-coder reliability or investigator triangulation examines whether different coders code the same data in a consistent way

©2010 John Wiley and Sons Chapter 11 Reliability measures Percent agreement Cohen’s Kappa: K = (Pa − Pc)/(1 − Pc) % Agreement = Number of cases coded the same way Total number of cases

©2010 John Wiley and Sons Chapter 11 Reliability check Agreement matrix

©2010 John Wiley and Sons Chapter 11 Reliability check Interpretation of Cohen’s Kappa

©2010 John Wiley and Sons Chapter 11 Subjective vs. objective coder Subjective coders – Knowledge and experience can help interpret the data – Less training required – May cause inflated reliability Objective coders – Less likely to cause inflated reliability – Lack of knowledge affect the ability to understand the data – More training required

©2010 John Wiley and Sons Chapter 11 Analyzing multimedia content The supporting techniques and methods are less mature compared to text data analysis Approaches: – Manual analysis: highly labor intensive and time consuming More accurate – Completely automated analysis: Faster, less amount of work Highly inaccurate – Partially automated approach Combines the advantages of the manual process and the completely automated process

©2010 John Wiley and Sons Chapter 11 End-of-chapter Summary Discussion questions Research design exercise