Achieving Appropriate Rigor in Qualitative Research Research Day, February 4, 2011 Mary Katherine O’Connor, Ph.D. School of Social Work, Virginia Commonwealth University

Presentation Goals Propose a multi-paradigmatic heuristic for understanding variety in qualitative research Propose a multi-paradigmatic heuristic for understanding variety in qualitative research Detail differential standards for research quality depending upon paradigmatic perspective Detail differential standards for research quality depending upon paradigmatic perspective Aid in matching research questions, analytical needs and computer-based qualitative analysis packages Aid in matching research questions, analytical needs and computer-based qualitative analysis packages

Two Dimensions of Analysis Subjective/objective relates to assumptions about the nature of knowing (and, therefore, of science) Subjective/objective relates to assumptions about the nature of knowing (and, therefore, of science) Regulation/radical change relates to assumptions about the nature of society (and, therefore, appropriate goals for science) Regulation/radical change relates to assumptions about the nature of society (and, therefore, appropriate goals for science)

Burrell & Morgan’s Framework FunctionalistInterpretive Radical Humanist Radical Structuralist Sociology of Regulation Sociology of Radical Change Objectivity Subjectivity

Multiple Paradigms Fundamentally different perspectives for analysis of social phenomena Fundamentally different perspectives for analysis of social phenomena Generating different concepts and analytical tools Generating different concepts and analytical tools Different standards for good research practice Different standards for good research practice Four different types of qualitative research building from different types of research questions and research goals Four different types of qualitative research building from different types of research questions and research goals

Multiple Paradigms in Qualitative Research

Interpretive Critical InterpretiveCritical Positivist Radical Change Regulation ObjectiveSubjective Positivist/ Post Positivist

Research Goals

Radical Change Regulation ObjectiveSubjective Positivist/ Post Positivist

Positivist Research: UNDERSTANDING FOR THEORY BUILDING THAT CAN RESULT IN THEORY TESTING Assumptions: Rational, pragmatic methods of natural science are appropriate to study human affairs Assumptions: Rational, pragmatic methods of natural science are appropriate to study human affairs Questions: Useful for theory building in the early stages of theory development Questions: Useful for theory building in the early stages of theory development Standards: understanding for generalizability including random assignment, control groups, valid and reliable data collection, standardized data analysis Standards: understanding for generalizability including random assignment, control groups, valid and reliable data collection, standardized data analysis

Critical Positivist Radical Change Regulation ObjectiveSubjective

Critical Positivist Research: CONSCIOUSNESS RAISING AND CHANGE AT THE SYSTEM LEVEL Assumptions: Knowledge is for radical/transformative change, emancipation and potentiality; realist but concentrating on structural relationships to understand and generate fundamental conflicts that will be the basis for radical change at the class/structural level Assumptions: Knowledge is for radical/transformative change, emancipation and potentiality; realist but concentrating on structural relationships to understand and generate fundamental conflicts that will be the basis for radical change at the class/structural level Questions: Focus on structural relationships seeking to provide explanations of the basic interrelationship within the context of total social formation for the purpose of consciousness raising Questions: Focus on structural relationships seeking to provide explanations of the basic interrelationship within the context of total social formation for the purpose of consciousness raising Designs: critical ethnographies, cooperative or collaborative inquiry, participatory action research, appreciative inquiry, empowerment evaluation Designs: critical ethnographies, cooperative or collaborative inquiry, participatory action research, appreciative inquiry, empowerment evaluation Standards: same as functionalist qualitative inquiry, but in addition, measurable presence of increased sophistication with change or change potential Standards: same as functionalist qualitative inquiry, but in addition, measurable presence of increased sophistication with change or change potential

Interpretive Radical Change Regulation ObjectiveSubjective

Interpretive Research: MEANING MAKING Assumptions: Understanding at the level of subjective experience from perspective of participants, not observer. Relativist in that the world, though ordered, is an emergent social enterprise that is continually being created. Assumptions: Understanding at the level of subjective experience from perspective of participants, not observer. Relativist in that the world, though ordered, is an emergent social enterprise that is continually being created. Questions: seeking explanation within individual consciousness and subjectivity from standpoint of the participants rather than the observer Questions: seeking explanation within individual consciousness and subjectivity from standpoint of the participants rather than the observer Designs: ethnographies, case studies, phenomenological studies, constructivist studies Designs: ethnographies, case studies, phenomenological studies, constructivist studies Standards: sampling for maximum variation, emergent design, multiple data collection approaches, triangulation, reflexivity, inductive data analysis, thick description of research product Standards: sampling for maximum variation, emergent design, multiple data collection approaches, triangulation, reflexivity, inductive data analysis, thick description of research product

Critical Interpretive Radical Change Regulation ObjectiveSubjective

Critical Interpretive Research: CONSCIOUSNESS RAISING AND CHANGE AT THE INDIVIDUAL LEVEL Assumptions: Critical interpretive with radical change focus, but from a subjectivist, individualistic standpoint. Concentrating on human consciousness and the social structures that inhibit true fulfillment. Goal is release of constraints that hamper individual human development. Assumptions: Critical interpretive with radical change focus, but from a subjectivist, individualistic standpoint. Concentrating on human consciousness and the social structures that inhibit true fulfillment. Goal is release of constraints that hamper individual human development. Questions: how individual makes sense of life and experiences of societal constraints Questions: how individual makes sense of life and experiences of societal constraints Designs: interview studies, narrative research, life histories, autoethnographies, collective biographies Designs: interview studies, narrative research, life histories, autoethnographies, collective biographies Standards: more artistic or aesthetic including creativity, aesthetic quality, interpretive vitality, and degree of stimulation for change Standards: more artistic or aesthetic including creativity, aesthetic quality, interpretive vitality, and degree of stimulation for change

Research Goals: Implications All qualitative research is not based on the same ontological and epistemological assumptions All qualitative research is not based on the same ontological and epistemological assumptions All qualitative research cannot be expected to be held to a universal standard of rigor, quality or worth All qualitative research cannot be expected to be held to a universal standard of rigor, quality or worth Key to justification of qualitative processes and products is selection of appropriate research design including data collection and analysis techniques able to answer research question lodged within specific paradigmatic perspective Key to justification of qualitative processes and products is selection of appropriate research design including data collection and analysis techniques able to answer research question lodged within specific paradigmatic perspectiveQUESTIONS?

Research Design

Positivist Design: An appropriate research design could include structured interviewing, focus group, grounded theory, etc. and should be identified and justified along with a data collection mechanism. An appropriate research design could include structured interviewing, focus group, grounded theory, etc. and should be identified and justified along with a data collection mechanism. The sampling plan is preferably randomized or at least looking for maximum variation along identifiable dimensions. The sampling plan is preferably randomized or at least looking for maximum variation along identifiable dimensions. An “n” of at least 30 or a justification for a lesser number that is similar to a justification for a lesser sample number in a quantitative project. An “n” of at least 30 or a justification for a lesser number that is similar to a justification for a lesser sample number in a quantitative project. A data analytic plan (thematic, content, constant comparison, linguistic, etc.) that matches the question, including whether or not computer based text analysis is being used. When computer analysis is present, the selected program must be able to do the analysis proposed. In most cases the variables for analysis should be identified beforehand. The exception would be in grounded theory designs. A data analytic plan (thematic, content, constant comparison, linguistic, etc.) that matches the question, including whether or not computer based text analysis is being used. When computer analysis is present, the selected program must be able to do the analysis proposed. In most cases the variables for analysis should be identified beforehand. The exception would be in grounded theory designs. Inter-rater reliability should also be addressed. Inter-rater reliability should also be addressed.

Critical Positivist Design: An appropriate research design could include empowerment, appreciative, focus group, etc. and should be identified and justified along with a data collection mechanism. An appropriate research design could include empowerment, appreciative, focus group, etc. and should be identified and justified along with a data collection mechanism. The sampling plan is preferably randomized or at least looking for maximum variation along identifiable dimensions. The sampling plan is preferably randomized or at least looking for maximum variation along identifiable dimensions. An “n” of at least 30 or a justification for a lesser number that is similar to a justification for a lesser number in a quantitative project. An “n” of at least 30 or a justification for a lesser number that is similar to a justification for a lesser number in a quantitative project. A data analytic plan (thematic, content, constant comparison, linguistic, etc.) that matches the question, including whether or not computer based text analysis is being used. When computer analysis is present, the selected program must be able to do the analysis proposed. In most cases the variables for analysis should be identified beforehand. A data analytic plan (thematic, content, constant comparison, linguistic, etc.) that matches the question, including whether or not computer based text analysis is being used. When computer analysis is present, the selected program must be able to do the analysis proposed. In most cases the variables for analysis should be identified beforehand. Inter-rater reliability should also be addressed. Inter-rater reliability should also be addressed. Change resulting from the research will need to be measured. Change resulting from the research will need to be measured.

Interpretive Design: An appropriate design could include ethnomethodological, constructivist, phenomenological, etc. and should be identified and justified. If the design is emergent, benchmarks for the emergence should be clear including how sampling and data collection will begin and might change. An appropriate design could include ethnomethodological, constructivist, phenomenological, etc. and should be identified and justified. If the design is emergent, benchmarks for the emergence should be clear including how sampling and data collection will begin and might change. Sampling for maximum variation is preferred and should identify the dimensions of variation of interest, the stakeholding groups to be sampled, and what might constitute saturation in determining the end of sampling and data collection. Sampling for maximum variation is preferred and should identify the dimensions of variation of interest, the stakeholding groups to be sampled, and what might constitute saturation in determining the end of sampling and data collection. A data analytic plan and the type of analysis should match the question and if computer-based analysis is being used, the program should be geared to interpretive analysis. Variables for analysis will rarely be identified beforehand; therefore, content analysis and constant comparison would be preferred. A data analytic plan and the type of analysis should match the question and if computer-based analysis is being used, the program should be geared to interpretive analysis. Variables for analysis will rarely be identified beforehand; therefore, content analysis and constant comparison would be preferred. Inter-rater reliability is not favored. Instead, member checking with participants is preferred. Inter-rater reliability is not favored. Instead, member checking with participants is preferred.

Critical Interpretive Design: An appropriate design could include an interview study, narrative research, life history, etc. and should be identified and justified. If the design is emergent, benchmarks for the emergence should be clear including how sampling and data collection will begin and might change. An appropriate design could include an interview study, narrative research, life history, etc. and should be identified and justified. If the design is emergent, benchmarks for the emergence should be clear including how sampling and data collection will begin and might change. Sampling will tend to be for a single case; but if the “n” is greater, maximum variation should identify the dimensions of variation of interest, the stakeholding groups to be sampled, and what might constitute saturation in determining the end of sampling and data collection. Sampling will tend to be for a single case; but if the “n” is greater, maximum variation should identify the dimensions of variation of interest, the stakeholding groups to be sampled, and what might constitute saturation in determining the end of sampling and data collection. A data analytic plan and the type of analysis should match the question and if computer-based analysis is being used, the program should be geared to interpretive analysis. Variables for analysis will rarely be identified beforehand; therefore, content analysis and constant comparison would be preferred. A data analytic plan and the type of analysis should match the question and if computer-based analysis is being used, the program should be geared to interpretive analysis. Variables for analysis will rarely be identified beforehand; therefore, content analysis and constant comparison would be preferred. Inter-rater reliability is not favored. Instead, member checking with participants is preferred. Inter-rater reliability is not favored. Instead, member checking with participants is preferred. Plans for documenting individual change as a result of the process can be noted. Plans for documenting individual change as a result of the process can be noted.

Data Analysis & Reporting

Paradigmatic Implications Content Analysis Content Analysis Thematic Analysis Linguistic Analysis Semiotics Constant Comparison Reporting ReportingCount Case Study Narratives

Content Analysis Thematic Analysis - reading “chunks” of data for meaning and attaching a meaning label (can then move to word count/distribution analysis) Thematic Analysis - reading “chunks” of data for meaning and attaching a meaning label (can then move to word count/distribution analysis) Linguistic Analysis - assessing word usage, message style (can then move to word count/distribution analysis) Linguistic Analysis - assessing word usage, message style (can then move to word count/distribution analysis) Semiotics - reading for meaning through use of metaphor (can then move to word count/distribution analysis) Semiotics - reading for meaning through use of metaphor (can then move to word count/distribution analysis) Constant Comparison - deconstruction and reconstruction through unitization and categorization (can then move to word count/distribution analysis) Constant Comparison - deconstruction and reconstruction through unitization and categorization (can then move to word count/distribution analysis)

Reporting Count- data display, distribution tables, conceptual or causal maps Count- data display, distribution tables, conceptual or causal maps Case Study – to chronicle (to record temporally and sequentially as in a history); to render (as in a description or to provide a vicarious experience); to teach (as instructional material); to test (using case as a trial for certain theories or hypotheses) Case Study – to chronicle (to record temporally and sequentially as in a history); to render (as in a description or to provide a vicarious experience); to teach (as instructional material); to test (using case as a trial for certain theories or hypotheses) Narrative – factual (he said/she said); interpretive (novel like); evaluative (elaborative judgments for evaluative testing) Narrative – factual (he said/she said); interpretive (novel like); evaluative (elaborative judgments for evaluative testing)

Computer Assisted Qualitative Data Analysis Software (CAQDAS) Misconceptions about software: A primary research instrument A primary research instrument A separate form of analysis A separate form of analysis Enhances rigor Enhances rigor Packages have similar paradigmatic assumptions Packages have similar paradigmatic assumptions

Data Analysis with CAQDAS Practical Considerations Available technology Available technology Skills and abilities Skills and abilities Time Time Political Considerations Institutional expectations (IRB, etc.) Institutional expectations (IRB, etc.) Peer review/professional/funding expectations Peer review/professional/funding expectations Paradigmatic Considerations Basic design assumptions Basic design assumptions Package similarities and nuanced differences Package similarities and nuanced differences

CAQDAS Disciplinary Backgrounds Atlas.ti – Interdisciplinary, Psychology, Linguistics, Computer Science Atlas.ti – Interdisciplinary, Psychology, Linguistics, Computer Science NVivo – Sociology NVivo – Sociology MAXqda – Political Science MAXqda – Political Science

CAQDAS Similarities Unitizing Unitizing Coding Coding Annotating Annotating Retrieving Retrieving Querying Querying Graphic representation Graphic representation Writing Support Writing Support

CAQDAS Differences Types of data Types of data Coding structure Coding structure Contextualization ContextualizationHowever…

Access + Awareness  Paradigmatic Congruence …When considering qualitative data analysis software:

Conclusion

When there is congruence between paradigmatic perspective, research goal and selected processes, articulation of what can and should be expected for accountability standards are possible and usefulness can be asserted. Positivist work cannot provide deep, individualized meaning, but can be basis for appropriately targeted quantitative work or provide evidence to support causal hypotheses. Positivist work cannot provide deep, individualized meaning, but can be basis for appropriately targeted quantitative work or provide evidence to support causal hypotheses. Critical Positivist work must engender class or structural changes from objective, generalizable data. Critical Positivist work must engender class or structural changes from objective, generalizable data. Interpretive work is not generalizable, but can provide complex, context-based deep understanding. Interpretive work is not generalizable, but can provide complex, context-based deep understanding. Critical Interpretive work must engender individual, subjective changes at the intersection of art, spirituality and scientific ways of knowing. Critical Interpretive work must engender individual, subjective changes at the intersection of art, spirituality and scientific ways of knowing.

Protocol Reviews (or…How do you defend what you are doing? Or assess student work?) Try to determine underlying assumptions in order to establish the paradigmatic location of the research by evaluating the aims or goals. Try to determine underlying assumptions in order to establish the paradigmatic location of the research by evaluating the aims or goals. Look for mixing of assumptions that suggest an incorrect understanding that qualitative research is always interpretive research Look for mixing of assumptions that suggest an incorrect understanding that qualitative research is always interpretive research Apply the standards for design and analysis quality consistent with the paradigm. Apply the standards for design and analysis quality consistent with the paradigm.

Words to the Wise for the Researcher Be aware of underlying assumptions in project that establish the paradigmatic location of the research, generally best articulated by project aims or goals. Be aware of underlying assumptions in project that establish the paradigmatic location of the research, generally best articulated by project aims or goals. Avoid mixing of assumptions that suggest an incorrect understanding that qualitative research is always interpretive research. Avoid mixing of assumptions that suggest an incorrect understanding that qualitative research is always interpretive research. Articulate and demonstrate the standards for design and analysis quality consistent with the paradigmatic perspective of the project…then choose your software Articulate and demonstrate the standards for design and analysis quality consistent with the paradigmatic perspective of the project…then choose your software