2. Agenda  2 Main QA approaches  Coding exercise 1  Coding exercise 2  Slides on Qualitative Analysis  Brainstorming Exercise (if time)  Affinity Diagramming Exercise (if time) 2

3. Qualitative Research: Common Features of Analytic Methods (Miles & Huberman,1994) 1 Affixing codes to a set of field notes drawn from data collection 2 Noting reflections or other remarks in margin 3 Sorting or shifting through the materials to identify similar phrases, relationships between themes, distinct differences between subgroups and common sequences

4. Qualitative Research: Common Features of Analytic Methods (Miles & Huberman,1994) 4 Isolating patterns and processes, commonalties and differences, and taking them out to the field in the next wave of data collection 5 Gradually elaborating a small set of generalisations that cover the consistencies discerned in the data base 6 Confronting those generalisations with a formalised body of knowledge in the form of constructs or theories

5. 2 general research approaches 5 deductive approach vs inductive approach

6. deductive research approach 6 THEORY HYPOTHESIS OBSERVATION CONFIRMATION Top-down approach Theory testing A priori codes

7. inductive research approach 7 THEORY TENTATIVE HYPOTHESIS PATTERN OBSERVATION bottom-up approach Theory building Emergent codes

8. 8 deductiveor inductive

9. Often use a hybrid approach  A set of a priori codes reflecting your understanding of the topic and your research questions  Emergent codes added as you code the data and find other factors/topics/codes that you had not considered 9

10. Exercise 1  Open coding  Inductive analysis  Exploratory research  Theory building research  http://b.socrative.com/login/student/ http://b.socrative.com/login/student/  Room: 7f156b7b 10

11. Exercise 2  Coding with pre-defined categories  Deductive analysis  Theory Testing  http://b.socrative.com/login/student/ http://b.socrative.com/login/student/  Room: 7f156b7b 11

13. Qualitative Inquiry - Purpose The purpose of qualitative inquiry is to produce findings. The Data Collection process is not an end in itself. The culminating activities of qualitative inquiry are analysis, interpretation, and presentation of findings.

14. Qualitative Inquiry - Challenge To make sense of massive amounts of data, reduce the volume of information, identify significant patterns and construct a framework for communicating the essence of what the data reveal

15. Qualitative Inquiry - Problem ‘…have few agreed-on canons for qualitative data analysis, in the sense of shared ground rules for drawing conclusions and verifying sturdiness’ (Miles and Huberman, 1984)

16. The Creativity of Qualitative Inquiry  ‘..the human element of qualitative inquiry is both its strength and weakness - its strength is fully using human insight and experience, its weakness is being so heavily dependent on the researcher’s skill, training, intellect, discipline, and creativity. The researcher is the instrument of qualitative inquiry, so the quality of the research depends heavily on the qualities of that human being’ (Patton, 1988)

17. The Science and Art of Qualitative Inquiry (Patton, 1988)  The Science The scientific part is systematic, analytical, rigorous, disciplined, and critical in perspective  The Art The artistic part is exploring, playful, metaphorical, insightful, and creative

18. 1. Analysis Considerations 1 Words 2 Context (tone and inflection) 3 Internal consistency (opinion shifts during groups) 4 Frequency and intensity of comments (counting, content analysis) 5 Specificity 6 Trends/themes 7 Iteration (data collection and analysis is an iterative process moving back and forth)

19. 2. The Procedures 1 Coding/indexing 2 Categorisation 3 Abstraction 4 Comparison 5 Dimensionalisation (relationships) 6 Integration 7 Iteration 8 Refutation (subjecting inferences to scrutiny) 9 Interpretation (grasp of meaning - difficult to describe procedurally)

20. Critical Thinking ‘Critical Thinking calls for a persistent effort to examine any belief or supposed form of knowledge in the light of the evidence that supports it and the further conclusions to which it tends’ (Glaser, 1941) or more simply! Critical Thinking means weighting up the arguments and evidence for and against.

21. Critical Thinking Key points (Glaser, 1941): –Persistence: Considering an issue carefully and more than once –Evidence: Evaluating the evidence put forward in support of the belief or viewpoint –Implications: Considering where the belief or viewpoint leads; what conclusions would follow; are these suitable and rational; and if not, should the belief or viewpoint be reconsidered

22. Guidance for Creative Thinking 1 Be open 2 Generate options 3 Divergence before convergence 4 Use multiple stimuli - triangulate 5 Side track, zig-zag, and circumnavigate 6 Change patterns of thinking 7 Make linkages 8 Trust yourself 9 Work and play at it

23. The Credibility of Qualitative Analysis 1 Rigorous techniques and methods for gathering high- quality data that is carefully analysed, with attention to issues of validity, reliability, and triangulation 2 The credibility of the researcher, which is dependent on training, experience, track record, status, and presentation of self 3 Philosophical belief in the phenomenological paradigm, that is, a fundamental appreciation of naturalistic inquiry, qualitative methods, inductive analysis and holistic thinking

24. A Credible Qualitative Study A credible qualitative study needs to address the following issues: 1 What techniques and methods were used to ensure the integrity, validity, and accuracy of the findings 2 What does the researcher bring to study in terms of qualifications, experience, and perspective 3 What paradigm orientation and assumptions ground the study

25. Principles of Analysing Qualitative Data 1 Proceed systematically and rigorously (minimise human error) 2 Record process, memos, journals, etc. 3 Focus on responding to research questions 4 Appropriate level of interpretation appropriate for situation 5 Time (process of inquiry and analysis are often simultaneous) 6 Seek to explain or enlighten 7 Evolutionary/emerging

26. Inter-rater reliability  What if you have more than one person coding?  How much agreement do they have?  At what point should you test their agreement?  Other than comparing counts, how can you validate the coding/analysis?  https://www.academia.edu/458025/The_place_of_inter- rater_reliability_in_qualitative_research_an_empirical_study https://www.academia.edu/458025/The_place_of_inter- rater_reliability_in_qualitative_research_an_empirical_study 26

27. 27 Articulate: who users are their key tasks User and task descriptions Goals: Methods: Products: Brainstorm designs Task centered system design Participatory design User- centered design Evaluate Psychology of everyday things User involvement Representation & metaphors low fidelity prototyping methods Throw-away paper prototypes Participatory interaction Task / Cognitive scenario walk-through Refined designs Graphical screen design Interface guidelines Style guides high fidelity prototyping methods Testable prototypes Usability testing Heuristic evaluation Completed designs Alpha/beta systems or complete specification Field testing Interface Design and Usability Engineering

28. brainstorming  the point is:  to generate MANY, WIDE-RANGING ideas  nutty and absurd are GOOD. go for the extremes (to get out of the rut)  riff off other’s ideas.  the point is NOT:  to generate excellent, complete, feasible ideas … pressure stifles  to develop or critique ideas … go wide. deep is for later.

29. process 1. prepare a list of topics / questions ahead of time; or in a preliminary brainstorm 2. facilitator takes team through list of topics switch topic when energy ramps down 3. Note taker takes notes (very important) 4. switch roles so everyone can play 5. ground rules 6. Follow up

30. brainstorming is like popcorn

31. ground rules  Postpone and withhold your judgment of ideas: never criticize  Encourage wild and exaggerated ideas  Quantity counts at this stage, not quality  Switch topics when the popcorn slows down  Build on the ideas put forward by others  Every person and every idea has equal worth  Elect a facilitator (calls switches) and a note- taker (one thought per post it!)

32. Post brain-storm  collect the notes  go through carefully, with judgment turned on  look for  interesting, surprising ideas that might work  ideas that will combine well  promising directions on which you should brainstorm more  keep your notes. at a later design stage, come back to them and see if anything else has become useful in the meantime.

33. Sometimes you have a lot of ideas to make sense of! 33

34. work consolidation: abstracting specific insights  one tool: the affinity diagram  can use to “consolidate” insights from collected or generated data. for example:  brainstorming about design problems  categories of problems  brainstorming about design ideas  categories of ideas  comments from users  categories of desirable / successful features

35. how do you make an affinity diagram? 1. team writes down all data & insights on post-it notes; be sure you can link the post-it detail back to its source! 2. stick one post-it on the wall a whiteboard or big sheet of butcher paper is best 3. arrange the other post-its around it, grouping by affinity to each other. iteration will be required. 4. look at each group and see what it has in common; name and describe each group. 5. “snapshot” the result for documentation digital photo  your design website or notebook transfer post-its onto paper, 1 sheet / notes-cluster  scan  website

36. why does an affinity diagram work? use physical arrangement/proximity to understand connections openness to serendipity low cost to rearrange ideas many variants:  arrange along axes rather than by affinity  tie causes to effects  group evidence under assertions

38. Pooya Jaferian, David Botta, Fahimeh Raja, Kirstie Hawkey, and Konstantin Beznosov. 2008. Guidelines for designing IT security management tools. In Proceedings of the 2nd ACM Symposium on Computer Human Interaction for Management of Information Technology (CHiMiT '08). ACM, New York, NY, USA,, Article 7, 10 pages. DOI=10.1145/1477973.1477983 http://doi.acm.org/10.1145/1477973.1477983 38

39. Methodology (Phase I) HOT Admin findings (4 papers) Guidelines from literature (14 papers) Set of guidelines (164 guidelines) Field studies Interviews Questionnaires Prototyping Cognitive walkthroughs Surveying other literature Field studies Interviews Questionnaires Prototyping Cognitive walkthroughs Surveying other literature Field study: Interviews Participatory observation Field study: Interviews Participatory observation 39

40. Methodology (Phase I) HOT Admin publications (4 papers) Guidelines from other literature (14 papers) Set of guidelines (164 guidelines) 40 Open Coding Axial Coding Categorized List of Guidelines

41. 41 High level Category Low level Category Guideline Guideline ID number

42. Methodology (Phase I) HOT Admin publications (4 papers) Guidelines from other literature (14 papers) Set of guidelines (164 guidelines) 42 Open Coding Axial Coding Affinity Diagrams Guidelines Framework

43. Methodology (Phase II) Additional 22 papers Broaden the literature review 5 Semi-structured interviews Not done solely for guidelines purpose Analyzed to find support for guidelines Illustration of guidelines with data from 5 interviews 43

44. Framework for classification of guidelines 44 Task Specific Organizational Complexity Technological Complexity Configuration and Deployment Diverse Stakeholders General Usability Guidelines Specificity Intensive Analysis Distributed ITSM Communication

45. 45 Framework for classification of guidelines Task Specific Guidelines General Usability Guidelines Technological Complexity Guidelines Make tools combinable [8,9,20,26]Use multiple levels of information abstraction [1,4,5,10,12,25,41,42,45] Help task prioritization [15,44]Use different presentation / interaction methods [1,4,5,29,41,48,49] Provide customizability [9,33]Support knowledge sharing [9,12,14,27,32,37,47] Technological Complexity Guidelines Make tools combinable [8,9,20,26]Use multiple levels of information abstraction [1,4,5,10,12,25,41,42,45] Help task prioritization [15,44]Use different presentation / interaction methods [1,4,5,29,41,48,49] Provide customizability [9,33]Support knowledge sharing [9,12,14,27,32,37,47] Organizational Complexity Guidelines Diverse Stakeholders Guidelines Provide flexible reporting [9,18,33,35] Provide an appropriate UI for stakeholders [9,35] Diverse Stakeholders Guidelines Provide flexible reporting [9,18,33,35] Provide an appropriate UI for stakeholders [9,35] Distributed ITSM Guidelines Support collaboration [6,7,20] Work in a large workflow [8,9,20] Distributed ITSM Guidelines Support collaboration [6,7,20] Work in a large workflow [8,9,20] Communication Guidelines Provide communication integration [6,7,28,45] Facilitate archiving [17,21 ] Communication Guidelines Provide communication integration [6,7,28,45] Facilitate archiving [17,21 ] Intensive Analysis Guidelines Provide customizable alerting [20] Provide automatic detection [26,41] Provide data correlation and filtering [1,26] Intensive Analysis Guidelines Provide customizable alerting [20] Provide automatic detection [26,41] Provide data correlation and filtering [1,26] Configuration and Deployment Guidelines Make configuration manageable [3,20] Support rehearsal and planning [3,6,7,20,44] Make configuration easy to change [20,46] Provide meaningful errors [20, 34,46] Configuration and Deployment Guidelines Make configuration manageable [3,20] Support rehearsal and planning [3,6,7,20,44] Make configuration easy to change [20,46] Provide meaningful errors [20, 34,46] More Specific

46. Class will be 1 big group 3 volunteer note takers  Problem:  How to design the user interface for a car proximity detection system  Brainstorm 3 aspects of the problem: (e.g., physical form factor, safety issues, input techniques, etc.)  go: 5 minutes

47. affinity diagram exercise  Now take your notes from the earlier brainstorming and create an affinity diagram  go: 8 minutes

48. debrief