1. What Can Qualitative Software Do for My Research? 2012 Summer Evaluation Institute June 4, 2012

2. Abt Associates | pg 2 Outline for the Day  Introductions and Purpose of the Course – 1:40-2:00  Brief Overview of Qualitative Research and Analysis – 2:00-2:25  Fun Coding Exercise – 2:25-3:10  BREAK – 3:10-3:30  Using Qualitative Software for Analysis - You drive! – 3:30-5:00

3. Abt Associates | pg 3 Brief Participant Introductions  Introduce yourselves to your tablemates  What do you hope to gain from today’s session? –Write down 3-4 ideas  One person at each table reports what you hope to gain

4. Abt Associates | pg 4 Introduction to Abt Associates  Mission-driven, global leader in research and program implementation in the fields of health, social and environmental policy, and international development  Strong experience in implementing complex qualitative and mixed methods studies in the United States and over 40 countries

5. Abt Associates | pg 5 Foundations of Qualitative Research and Analysis  Theoretical underpinnings of qualitative research methods  Evolution and functionality of qualitative analysis software  Future directions in qualitative research

6. Abt Associates | pg 6 What is Qualitative Data? What is a Qualitative Method?  Type: Any non-numeric, or text-based data –Verbal data –Visual data –Audio data –Observational data –Written words  Data Collection Methodology: A way of systematically collecting qualitative data  Data Analysis: A way of systematically analyzing a qualitative dataset

7. Abt Associates | pg 7 Types of Qualitative Data/Methods  Primary Data Collection –In-depth interviews (problem-centered, expert, elite, ethnographic, narrative) –Focus Groups and group interviews/discussions (dyads, triads, joint narratives) –Direct observation –Case studies (use the methods listed above, usually in combination)  Primary and Secondary Data Collection –Text from written documents –Visual images from photos, film/video, art, performance –Audio recordings –Web-based discussions (chat rooms, remotely-facilitated discussions)

8. Abt Associates | pg 8 What Determines the Research Method? The research (or evaluation) questions: What kind of data can shed light on the question? The state of knowledge (e.g., qualitative methods are often used during early stages of inquiry or to explore quantitative findings) The purpose of the research: to develop or test hypotheses, to explain a phenomenon, to gain insight, or to develop a typology The audience (e.g., client) that will use, assess, judge, or take action on the results The resources available to support the inquiry (e.g., timeframe, funding) The characteristics of the study population and feasibility of recruitment

9. Abt Associates | pg 9 Example  A client has asked Abt to investigate why there are large regional variations in the contraceptive prevalence rate in the Philippines. Even though there have been high levels of national and donor investment in promoting family planning (FP) uptake in the Philippines over the last ten years, some regions appear to have made larger gains in FP utilization than others. Why?  What information do you need to know to determine the research method?

10. Abt Associates | pg 10 Hallmarks of Qualitative Methods  Certain questions lend themselves to qualitative methods  Facilitate study of issues in detail and depth  Useful in developing relevant survey questions  Not constrained by pre-determined measures and categories of analysis  Add nuance and complexity to quantitative results  Usually not generalizable: Often involve a small number of cases, but researchers do utilize qualitative methods in large scale projects

11. Abt Associates | pg 11 Threats to Reliability and Validity What is Validity?  The extent to which the research produces an accurate version of the world  Internal validity – the degree to which the investigator’s conclusions correctly portray the data collected  External validity – the degree to which conclusions are appropriate to generalize to similar populations/locations outside the study area What is Reliability?  The extent to which research produces the same result when replicated  Some argue that reliability is an impossible criterion to achieve as different researchers will always produce different versions of the social world

12. Abt Associates | pg 12 Promoting Reliability and Validity  Consider mixed method approaches  Document the process of analysis including what is the statement of a respondent and what is interpretation by a researcher  Involve multiple analysts to check biases  Document in detail the process by which analytical themes or codes are developed  Train coders or analysts on coding structure and create well- defined themes. Refine again and again.  In analysis, check inter-rater reliability  Develop conventions for transcribing data so that transcripts are comparable across data source  Develop saturation guidelines

13. Qualitative Analysis Software: Introducing QSR International

14. Introduction to QSR International Established in 1995 to further the development of the NUD*IST software originally developed in 1981 Currently has over 400,000 users in 150 countries Americas regional office for Training and Consulting is located in Cambridge, Massachusetts and Technical Support services are located at corporate headquarters in Melbourne, Australia

15. A Brief History of CAQDAS Computer Assisted Qualitative Data Analysis Software 1981 – Lyn and Tom Richards develop NUD*IST, the precursor to NVivo 1994 - Miles and Huberman discuss the use of software in qualitative analysis in their widely cited text 2007 – National Science Foundation publishes guidelines for the use of software in qualitative data analysis 2012 - NVivo, AtlasTi, EZ-Text, ANSWR, MaxQDA, HyperResearch and Dedoose are among the most commonly used tools today

16. How can Software Help Improve the Rigor of Qualitative Analysis? GATHER Existing Research Current Information Data Sources ORGANIZE Data Sources Coding Structure Demographics Literature Review ANALYZE Coding Queries Memos Visuals VISUALIZE RESULTS Reports Journal Articles Dissertations Study presentations Transparency Saturation Methodology

17. The Analysis Process Text Search Word Frequency Matrix Coding Comparison Models Matrices Charts Key ideas Reflect Link Review Merge Refine CodingMemoing QueryingVisualizing

18. Visual Options Charts Models Cluster Analysis

19. Mixed Methods with NVivo

20. Abt Associates | pg 20 Considerations in Choosing CAQDAS like NVivo  Sample size and multiplicity of data sources  Emphasis on replicability, rigor and transparency  Likelihood that there will be future opportunities to perform secondary analyses on the same dataset  Desire to publish in peer-reviewed journals  Interest in merging close-ended attributes into the qualitative dataset  Building capacity of analysis team including training time and costs  Budgetary parameters and software investment

21. Abt Associates | pg 21 Future Directions in Qualitative Analysis  Mixed methods tools, such as the capacity to work with datasets containing both fixed response and open-ended material  Web-based data, including social media and online discussion boards  Capacity for larger samples, especially large quantities of text (qualitative studies are no longer small)  More tools for comparing coding by researcher, theme, and participant group

22. Descriptive Statistics Inferential Statistics Meta-Analysis Coding of Text Audio, Video and Image Data Open-Ended Survey and Interview Responses Transcribing tools for Audio and Video Import/Export from Excel, text and database files Open-Ended and Fixed Response Questions Within and Between Group Analysis of Coding Text Analysis Kappa Coefficient Cluster Analysis of Word and Coding Similarity QualitativeQuantitative NVivo A Future Look at Mixed Methods with NVivo

23. Abt Associates | pg 23 Contact Information  Lisa LeRoy, MBA, PhD, lisa_leroy@abtassoc.comlisa_leroy@abtassoc.com  Ilana Ron, MSc, ilana_ron@abtassoc.comilana_ron@abtassoc.com  Hilary Rhodes, PhD, hilary_rhodes@abtassoc.comhilary_rhodes@abtassoc.com  Cynthia Jacobs, EdD, C.Jacobs@qsrinternational.com C.Jacobs@qsrinternational.com

24. Abt Associates | pg 24 Brief Coding Exercise - Instructions  Read the interview with Thomas – 10 minutes  Remember the purpose of the research  Work in pairs. Summarize the main points in the text. Put parentheses around the text and write the word/phrase that summarizes the text next to it.  Note questions or ideas that occur to you as you read the interview.  Report out and discussion – 25 minutes

25. Abt Associates | pg 25 Brief Coding Exercise – Discussion  What codes did you and your partner come up with?  What codes are similar to each other?  How will we capture the changing definitions over time?  How will we merge codes?  How will we assess inter-rater reliability?  What questions or comments occurred to you as you read the interview?

26. Abt Associates | pg 26 Brief Coding Exercise – Summary  What works manually with a small amount of qualitative research becomes more complicated as the number of qualitative sources increase  Defining, redefining, merging, and separating codes is easier to do and easier to keep track of with qualitative software  Documenting the analysis process systematically is a benefit of the software; replicability is possible

27. Questions and Comments?

28. Using NVivo to Facilitate Qualitative Data Analysis

29. NVivo in Practice  Increase accessibility of data and transparency of analysis  Identify and test ideas about emerging patterns and themes  Utilize open-ended text and non-text data  Generate output for reporting  Support team-based analysis

30. How do respondents describe the environmental changes down east?