Policy Relevance of Increased Rigor in Qualitative Methods: The Case of Experimental Evaluations in Education Hannah Betesh Berkeley Policy Associates bpa

Goals of the session Invite the qualitative research community to embrace transparency in analysis as one form of rigor (‘showing’ as well as telling our work) Unpack our team-based analytical approach Translate a theoretical big idea into the nuts and bolts of qualitative analysis

Policy Context Emphasis on accountability under No Child Left Behind Increased use of “medical model” in evaluation of educational programs. Heightened focus on quantitative outcomes, to the exclusion of qualitative findings on implementation.

Research Context Very recent emergence of qualitative tradition in policy research (Rist 1994) Qualitative research often impugned by many policy researchers as insufficiently valid and/or reliable (Howe & Eisenhart 1990) There is, sometimes, a perceived lack of ‘rigor’ in qualitative educational research (Guba 1981)

Importance of Rigorous Qualitative Analysis in Policy Research Mixed-methods research can generate richer findings than either qualitative or quantitative analysis alone (Creswell and Plano Clark 2007) Qualitative inquiry can help unpack implementation issues to inform quantitative findings Capture perspectives of marginalized populations to elucidate differences in quantitative data (Tillman 2009)

Transparency as Rigor The ways in which researchers “account for and disclose their approach to all aspects of the research process are key to evaluating their work substantively and methodologically” (Anfara, Brown, and Mangione, 2002).

Facets of Rigorous Qualitative Research Transparent and public coding scheme Interim research products to track evolution of analysis Explicit links to any quantitative component of the research

Components of Rigorous Analysis Team-based data analysis Use of software Multi-stage analysis process Laying a strong foundation

Components of Rigorous Analysis Team-based data analysis

Components of Rigorous Analysis Identifying salient conceptual categories Team-based data analysis

Components of Rigorous Analysis Identifying salient conceptual categories Examining relationships Team-based data analysis

Components of Rigorous Analysis Identifying salient conceptual categories Examining relationships Reliability-enhancing mechanisms Team-based data analysis

Components of Rigorous Analysis Salient conceptual categories Examining relationships Reliability-enhancing mechanisms Going beyond code and retrieve Team-based data analysis

Data Research questions Identifying Salient Categories

What Now? Study description Specific application of process Software demonstration