1. Working with Qualitative Data Christine Maidl Pribbenow Wisconsin Center for Education Research cmpribbenow@wisc.edu

2. Session Outline General discussion about educational research, assumptions and misconceptions Contrast educational research with research in the sciences Define common qualitative analysis terms Provide example using ATLAS.ti– qualitative analysis software program Code some text

3. Free Association…

4. DATA

5. QUALITATIVE

6. Qualitative Data: Oxymoron or inherent tensions? Hard vs. soft (mushy) Rigor Validity and reliability Objective vs. subjective Numbers vs. text What is The Truth?

7. What are some of the assumptions that you have about educational research? How are they helping or hindering the development of your study?

8. Research in the sciences vs. research in education “Soft” knowledge Findings based in specific contexts Difficult to replicate Cannot make causal claims due to willful human action Short-term effort of intellectual accumulation– “village huts” Oriented toward practical application in specific contexts “Hard” knowledge Produce findings that are replicable Validated and accepted as definitive (i.e., what we know) Knowledge builds upon itself– “skyscrapers of knowledge” Oriented toward the construction and refinement of theory

9. Strongly Agree AgreeUnsureDisagreeStrongly Disagree Educational research is rigorous.4 (31 %)8 (62 %)1 (8 %)0 (0 %) I have read at least ten articles published in educational research journals before attending this Institute. 11 (85 %)2 (15 %)0 (0 %) Educational research is more difficult than my scientific research. 1 (8 %)2 (15 %)6 (46 %)3 (23 %)1 (8 %) I regularly collect qualitative data in my classes for assessment purposes. 1 (8 %)5 (38 %)2 (15 %)4 (31 %)1 (8 %) I need a control or comparison group to conduct educational research. 2 (15 %)3 (23 %)1 (8 %)5 (38 %)2 (15 %) Assessment data gleaned from students (i.e., "self report") are valuable. 2 (15 %)9 (69 %)2 (15 %)0 (0 %) I have analyzed qualitative data in the past. 1 (8 %)3 (23 %)1 (8 %)4 (31 %)

10. Strongly Agree AgreeUnsureDisagreeStrongly Disagree Qualitative data can meet "reliability" standards. 2 (15 %)5 (38 %)6 (46 %)0 (0 %) Qualitative data can meet "validity" standards. 2 (15 %)5 (38 %)6 (46 %)0 (0 %) If I collect learning assessment data from my students and the analyzed results are "not significant" it proves that students did not learn what I intended. 0 (0 %) 1 (8 %)5 (38 %)7 (54 %) If I conduct classroom research and the results are "not significant", the study was a waste of my time. 0 (0 %) 1 (8 %)3 (23 %)9 (69 %) I need human subjects approval to conduct and publish research about my students. 7 (54 %)1 (8 %)5 (38 %)0 (0 %) I want to conduct research in my classroom so that I can teach better. 11 (85 %)2 (15 %)0 (0 %) I want to conduct research in my classroom so that my students learn more or better. 13 (100 %)0 (0 %)

11. What are some sources of qualitative data? Lab notebooks Open-ended exam questions Papers Journal entries On-line discussions Email Notes from observations

12. Qualitative Data Analysis Qualitative analysis is the “interplay between researchers and data.” Researcher and analysis are “inextricably linked.”

13. Qualitative Data Analysis Inductive process –Grounded Theory Unsure of what you’re looking for, what you’ll find No assumptions No literature review at the beginning Constant comparative method Deductive process –Theory driven Know the categories or themes using rubric, taxonomy Looking for confirming and disconfirming evidence Question and analysis informed by the literature

14. Example Research Questions Why do faculty leave UW-Madison? Do UW-Madison faculty leave due to climate issues?

15. Definitions: Coding and Themes Coding process: –Conceptualizing, reducing, elaborating and relating text– words, phrases, sentences, paragraphs. Building themes: –Codes are categorized thematically to describe or explain phenomenon.

16. Let’s Code #1 Read through the reflection paper written by the student from an Ecology class and highlight words, parts of sentences, and/or whole sentences with some “code” attached and identified to those sections.

17. What did you highlight? Why?

18. Let’s Code #2 Read through this reflection paper and code based on this question: What were the student’s assumptions or misconceptions before taking this course?

19. What did you highlight? Why?

20. Let’s Code #3 Read through this reflection paper and code based on this question: What did the student learn in the course?

21. What did you highlight? Why?

22. Can we say that the students learned something in the course using reflection papers? Why or why not?

23. Ensuring “validity” and “reliability” in your research Use mixed methods, multiple sources. Triangulate your data whenever possible. Ask others to review your design methodology, observations, data, analysis, and interpretations (e.g., inter-rater reliability). Rely on your study participants to “member check” your findings. Note limitations of your study whenever possible.

24. Does the redesign of an ecology course to include concept maps derived from current journal articles help students to gain a more current and realistic view of ecological issues?

25. Questions?

26. The plural of anecdote is data. -Donna Shalala