Exploring the Roles of Faculty Supervision: Improving Qualitative Doctoral Dissertation Methodology Dan Kaczynski, PhD

Research Problem Increasing emphasis on quality Understand assessment practices Strengthen qualitative research skills Develop future researchers More qualitative dissertations Shifting supervisory roles

Open Discussion  Do you use technology in your research?

Open Discussion  How should we explore the tensions within and between:  Assessing Quality  Adoption of QDAS

Qualitative Software NVivo, MAXqda, Atlas ti, QDA Miner, Qualrus, Transana Kaczynski (2004)

What is Good Qualitative Research?

What does good work look like? Identify indicators of quality in a thesis:       Identify common errors in a thesis:      

What is Quality? The researchers logic of justification The researchers logic of justification “Flaws in the logic of justification can potentially occur anywhere in the inquiry process. The nature of such flaws and where they occur can jeopardize the soundness of a study in one or more ways.” “Flaws in the logic of justification can potentially occur anywhere in the inquiry process. The nature of such flaws and where they occur can jeopardize the soundness of a study in one or more ways.” (Piantanida & Garman 1999, p. 147)

Types of Quality Criteria  Philosophical Criteria (Lincoln, 1995; Lincoln & Guba, 1985)  Procedural Criteria (Creswell, 1998)

Philosophical Criteria (Lincoln & Guba, 1985)  Credibility  Is the work authentic?  Transferability  Will the work fit outside this situation?  Dependability  Is the researcher consistent?  Confirmability  Are interpretations defensible?

Procedural Criteria  Quality of methods (open-ended interviews)  Quality of data (verbatim long transcripts)  Quality of data analysis (comprehensive data treatment) (Silverman, 2004 [Sacks, 1984])

Standardized Procedural Criteria (controversial checklists or guidelines)  Does the title reflect the study focus?  Is the problem socially important?  Is the literature review comprehensive?  Has study conformed to ethics standards?  Are issues of sampling discussed?  Did findings answer the questions?  Was study written convincingly?  Are issues of trustworthiness addressed?

Quantitative and Qualitative Criteria for Assessing Research Quality and Rigor Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002) p. 30 Quantitative term Qualitative term Strategy employed Internal validity Credibility  Prolonged engagement in field  Use of peer debriefing  Triangulation  Member checks  Time sampling External validity Transferability  Provide thick description  Purposive sampling ReliabilityDependability  Create an audit trail  Code-recode strategy  Triangulation  Peer examination ObjectivityConfirmability  Triangulation  Practice reflexivity

Transparent Assessment  Explore rich diversity of meanings  Sensitized appreciation of worth  Deeper assessment of analysis  Multiple paths to look inside  Transparency strengthens credibility

Data Collection Stage 1 Stage 2 Stage 3 Survey Research Interviews/ Documents Document Review Supervisory Ability Quality Factors Technology Resources Professional Development PerceptionsProcesses Assessment Practices

Stage 1 Survey Questions  What are faculty members’ attitudes toward qualitative research?  How do faculty members conceptualize quality in doctoral level qualitative research?  What processes do faculty members use as they work through qualitative methodology issues with dissertation students?  How do faculty members assess the quality of qualitative and mixed-methods dissertations?  What technological tools do faculty members use to review student research?  What are faculty members’ perceptions of the skills needed to assess qualitative research?  What professional developments needs exist among faculty for evaluating qualitative research?

Findings: Knowledge, Ability, and Confidence Satisfactory or Higher MSD Supervising73.9% Serving on a committee 91.3% Judging quality 91.3%

Findings: Technology Tools Used in Assessment 52%(None) 30%(Not applicable) 17%NVivo 4%InfoRapid

Findings: Resources Consulted for Expertise 83%Others 70%Publications 30%Conference workshops 22%Campus workshops 13% Continuing education

Findings: Conceptualizations of Quality (cont.) AIAlternative interpretations AI: Alternative interpretations CE: Consideration of ethical issues AG: Ability to generalize findings HC: Hierarchical code structure MC: Member-checking MSMemos MS: Memos AT: Methodological audit trail PD: Peer debriefing PF: Prolonged field engagement AS: Qualitative data analysis software ROResearcher objectivity RO: Researcher objectivity SSSampling strategies SS: Sampling strategies SDSelf-disclosure SD: Self-disclosure SC: Social context TOTheoretical orientation TO: Theoretical orientation TN: Triangulation VYValidity VY: Validity

Stage 2 Findings: Critical Needs  Building knowledge and skills –Moving beyond superficial assessment –Significance of researcher transparency –Teaching students to self-assess  Building a community of practice –Strengthening qualitative research skills –Sharing assessment strategies –Engaging in professional development

Stage 3 Findings: Role of Technology

Study Findings  Highly favorable attitudes toward qualitative research  Diverse conceptualizations of quality  Need for alternative assessment frameworks  Need and desire to strengthen knowledge and skills  Need for professional development

Future Challenges  Progressing research methods Mixed → Blended → Integrated  Emerging research innovations  mainstream adoption of QDAS  Positioning quality research standards

Future Research Questions  What does it mean to disclose or conceal the role of technology?  Does nondisclosure of analysis software imply the presumption that the use of technology is ubiquitous and commonly accepted?  Does nondisclosure of QDAS suggest a student’s fear of the supervisor’s acceptance or sanctioning?  In what ways and under what conditions does a technological tool become a barrier to the learning process for the teacher and the learner?