Incorporating NVivo into a Large Qualitative Project Using NVivo software as part of a multi team analysis of in depth interview data in a mixed methods evaluation Karla Eisen, MSW, MPH Izabella Zandberg, PhD Susan Berkowitz, PhD

Overview Project overview Approach and structure of qualitative analysis Incorporating NVivo Different approaches of analytic teams Lessons learned 2/18/2019

Project Overview IECRN – a large scale mixed method study of clinical research networks (CRNs) in the United States and internationally. Multiple Components Descriptive Survey Component- detailed description of characteristics of a purposive sample of clinical research networks (CRNs); modules for different practice domains. Original sample of 133/response rates varied by module. 2/18/2019

Project Overview : Descriptive Study Qualitative Component: in-depth, semi-structured interviews to discern barriers and facilitators of CRNs’ efficiency and effectiveness in 6 domains of network functioning. Management and Governance Training and Professional Development Recruitment and Retention Network Operations Information Technology Data Management Quantitative Component 2/18/2019

Qualitative Data Collection 6 surveys with qualitative component Range of respondents: Principal Investigator, Administrative Manager, Data Manager, etc. Interview teams grouped by domain clusters Varying level of experience in qualitative interviewing Quantitative data retrieval mixed with some qualitative interviews Interviews recorded and transcribed (transcripts) Interviews summarized (summary files). 2/18/2019

Approach to Qualitative Analysis One qualitative analysis group of nine analysts Separate analytic teams clustered by domain area Bi weekly meetings to discuss emerging themes, cross domain analysis, etc. Analysts from multiple sites, e-room used Separate NVivo databases Separate report sections by analytic teams merged into one report NVivo training. 2/18/2019

NVivo Training First training Second training Far in advance of data collection period Incorporated day long overview on qualitative research (to prepare for qualitative interviewing and emphasize theoretical approach). Second training Very close to start of analysis Used actual project data collected Basic coding tree already developed Ongoing support via consultation with trainer available throughout analysis period. 2/18/2019

Structure of Analysis Analysis unique to domains, but similar in structure Question Areas (unique) Barriers and facilitators (shared) Major concepts and categories (partially shared) Basic coding tree developed and refined through discussion within analyst group Further code refinement anticipated within each analytic team Tension between “pre set” coding categories and unique content. 2/18/2019

Different Approaches to Qualitative Analysis with NVivo: Approach 1 Proximity to data Two out of three members of the team conducted the interviews they were analyzing. One team member developed  interview instruments and trained  interviewers on their use.  Good data quality (all transcripts) Coding structure developed (starting pre-NVivo) and revised iteratively Regular meetings to discuss the analytic process, emerging themes, code refinement, etc. Different levels of involvement with the software Team members highly motivated to learn NVivo NVivo tools used to guide analysis (Merge, Node Reports, and Matrix Intersections). 2/18/2019

Different Approaches to Qualitative Analysis with NVivo: Approach 2 Proximity to data Analysts not directly involved in interviewing Some involvement in developing interview instruments. Data quality inconsistent Wide range of analytic skills and experience within the team Used basic coding structure Different levels of involvement with the software Used NVivo for rough data categorization (node reports) Approached NVivo more quantitatively (focus on inter coder reliability, quality checks, etc.). 2/18/2019

Different Approaches to Qualitative Analysis with NVivo: Approach 3 Proximity to data Analysts mixed involvement in interviewing Data quality inconsistent Analysts worked primarily alone Used basic coding structure Used NVivo to code and organize text, but did not take advantage of other software functions Additional analysis done traditionally “by hand.” 2/18/2019

Different Approaches to Qualitative Analysis with NVivo: Approach 4 Proximity to data Analysts not directly involved in interviewing One team member helped revise and train on instruments.  Data quality marginal Wide range of analytic skills and experience within the team (Senior analyst and content expert) Team dissolution Limited use of coding structure NVivo not used Analysis conducted solely using traditional “pencil and paper” methods. 2/18/2019

Lessons Learned Challenge of Diverse Analytic Teams Experience with instrument design and conducting qualitative interviews Qualitative analysis experience/abilities Quality of qualitative data to analyze Prior exposure to NVivo or other qualitative analysis software packages Motivation to learn software and use in analysis process Staff from multiple companies, multiple divisions within companies (expectations?) 2/18/2019

Lessons learned (continued) Proclivity to use NVivo related to experience and abilities of analysts, team make up and support, and comfort with software as an analytic tool Learning curve (NVivo) Start up time, qualitative data management issues Time–pressured, deadline-driven non-academic environment Full potential of the software unexplored Unable to successfully integrate attributes from quantitative data. 2/18/2019

Next Time Around………… More review of coding consistency within teams and across teams More practice running reports (matrix intersection “cross tabs” and attributes) More use of NVivo features (attributes, memos, etc.) More practice moving from NVivo reports to meaningful finished text More discussion of emerging themes across groups More time for writing, editing, and rewriting report. 2/18/2019

Contact:  Karla Eisen WESTAT 1650 Research Blvd. Rockville, MD 20850-3129 KarlaEisen@westat.com (301) 315-5927 2/18/2019