1. AVI/Psych 358/IE 340: Human Factors Data Analysis October 22-24, 2008

2. 2 What we learned (Review) Principles of design –Usability and user experience –Human cognition (perception, visual design) –Interface design principles (metaphors, affordances) Requirements –From conceptual model to functional requirements –Data gathering techniques

3. 3 What we learned (Review) Prototyping –Low fidelity (paper, cardboard etc) –High fidelity

4. 4 What’s Ahead What to do with data –Qualitative analysis –Quantitative analysis Conducting evaluations –Formative evaluation –Summative evaluation

5. 5 Overview  Qualitative and quantitative  Simple quantitative analysis  Simple qualitative analysis  Tools to support data analysis

6. 6 What to do with data (1/2) How can you interpret the data gathered from surveys, think aloud protocols or questionnaires? Two types of data –Qualitative –Quantitative

7. 7 What to do with data (2/2) Interviews –Responses (qualitative), demographic information (quantitative) Questionnaires –Responses (qualitative), demographic information (quantitative) Observation –Notes, audio, video recordings, data logs, think- aloud

8. 8 Quantitative analysis Quantitative data – expressed as numbers Data from surveys (e.g., likert scale type of surveys), logs (e.g., mouse clicks), demographic data Quantitative analysis – numerical methods to ascertain size, magnitude, amount

9. 9 Simple quantitative analysis Averages –Mean: add up values and divide by number of data points –Median: middle value of data when ranked –Mode: value that appears most often in the data Percentages

10. 10 Quantitative visualizations Graphical representations give overview of data

11. 11 More advanced analysis Regression: relationship between variables (prediction) - –E.g. relationship between variables y, x, and z can be expressed as: y = ax + bz + c (a, b, c are constants) Correlation ANOVA (analysis of variance) Chi-square (testing independence between two variables) Time series analysis

12. 12 Qualitative analysis Qualitative data – difficult to measure sensibly as numbers, e.g. count number of words to measure dissatisfaction Requires careful analysis, thought and interpretation

13. 13 Types of qualitative data Unstructured - are not directed by a script. Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability

14. 14 Simple qualitative analysis Recurring patterns or themes –Emergent from data, dependent on observation framework if used Categorizing data –Categorization scheme may be emergent or pre-specified Looking for critical incidents –Helps to focus in on key events

15. 15 In-class activity (Part I) Brainstorm functional requirements for a next generation digital camera –Write each requirement on a sheet of paper (come up with at least 15 enhancements) –Put all notes on your desk

16. 16 In-class activity (Part II) Categorize the functional requirements –Organize notes into groups based on similarities. Note that groups must emerge from the requirements you identified. –Label each group of notes with a name (e.g., picture capabilities, size and feel)

17. 17 Affinity Diagram

18. 18 Tools to support data analysis Spreadsheet – simple to use, basic graphs Statistical packages, e.g. SPSS Qualitative data analysis tools –Categorization and theme-based analysis, e.g. N6 –Quantitative analysis of text-based data CAQDAS Networking Project, based at the University of Surrey (http://caqdas.soc.surrey.ac.uk/)

19. 19 Sources of Qualitative Data Interviews (structured, semi-structured and unstructured), Focus groups Think-aloud walkthroughs Questionnaires (open ended) Participant observation notes

20. 20 Categorizing qualitative data Challenges –categorizing the data in a meaningful way –Tedious (requires a lot of patience) –Interpretation Analysis through “coding”: marking the data and categorizing it Two basic types: –Structured coding (pre-defined coding scheme) –Emergent or open coding (Grounded Theory)

21. 21 Structured coding Apply pre-defined codes to data Where do the codes come from? –Theory –Literature, Other studies

22. 22 Example of structured coding In-class handouts Using think-aloud, users are working in an online educational setting using different navigation aids

23. 23 Structured coding activity (Part I) Using the same coding scheme in the example just shown, code the transcript on page 4 of your handout. You can underline, circle, highlight, or use any other way to code the text (don’t be restricted by the technique of using square brackets).

24. 24 Structured coding activity (Part II) Compare and contrast your coded data with the person next to you. Are there differences? Are there similarities? Are the exact same chunks of text coded? Etc.

25. 25 Emergent or open coding Aims to derive theory from systematic analysis of data (Grounded Theory) –Categories are discovered in the data from the data No pre-defined coding scheme

26. 26 Example of open coding How does collaboration happen in virtual teams? Chat transcripts were obtained and coded

27. 27 Example of open coding for excerpt of chat transcript Hi there. I’m Henry. I just wanted to say hello first and provide you with the rest of our group members’ email addresses. (Names and email addresses). Well, I guess we’ll see each other virtually on Saturday using the videoconferencing system. For coding, use: –Codes –Memos (notes)

28. 28 Example of open coding for excerpt of chat transcript Hi there. I’m Henry. I just wanted to say hello first and provide you with the rest of our group members’ email addresses. (Names and email addresses). Well, I guess we’ll see each other virtually on Saturday using the videoconferencing system. Leadership: initiative to represent Preference for communication technology: implying that email may be appropriate Co-presence: establishing team’s virtual presence Advanced technology can bridge time and space: videoconferencing can help to coordinate synchronous meetings

29. 29 Open coding activity Code the chat transcript on page 5 of your handout using open coding. You can underline, circle, highlight, or use any other way to code the text.

30. 30 Issues in “coding” Inter-rater reliability –Get two independent coders to code data –Calculate reliability (percentage of agreement) between the coders Granularity –At what levels are the codes being applied –Words, line, sentence, conversation, etc.

31. 31 Presenting the findings Only make claims that your data can support The best way to present your findings depends on the audience, the purpose, and the data gathering and analysis undertaken Graphical representations may be appropriate (e.g., the frequency of occurrence of codes) Other techniques are: –Rigorous notations, e.g., UML –Using stories, e.g., to create scenarios –Summarizing the findings

32. 32 Summary The data analysis that can be done depends on the data gathering that was done Qualitative and quantitative data analysis Two basic types of coding