1. Teaching “Qual” inside a “Quant” World John F. Stevenson November 12, 2010

2. Setting the Context  One class out of thirteen in a stand-alone graduate course on program evaluation  Psychology Department doctoral program students  Primary orientation: Quantitative; Campbell  Logic model framework organizes much of the course

3. Rationale for a Qualitative Component  Campbell’s “Qualitative Knowing”  Value of reflection in any evaluation context  Knowing what you aren’t seeing and what you may not know  Multicultural theme for the course  Invocation of dialogue, alternative perspectives, from the beginning of the semester

4. The Course in Brief  Evaluation theory and paradigm concerns; “other voices” in evaluation  The cycle: Needs & Resources; Plan; Implementation; Evaluation; Feedback loops along the way  Planning an evaluation: Program theory and the logic model  Needs assessment  Process and implementation evaluation  Measurement  Outcome evaluation  Efficiency evaluation  Evaluation reporting and utilization

5. The Qualitative Day  Introduction: Groundrule negotiation  Setting the scenario: A fictional evaluation of the multicultural competence of our graduate programs’ training  First exercise: Shaw’s “Cultural Review” exercise  List topics for self-inquiry related to this project  Write a paragraph “unpacking” one topic  Share voluntarily

6. The Qualitative Day  Second exercise: Participatory research planning role play  Assign stakeholder roles to students (program directors, department chair, students from each program)  Describe the task for the planning committee  What questions would you ask?  What methods would you choose?  Instructor leaves the room to let the drama unfold  Instructor returns to learn what the plan is

7. The Qualitative Day  Qualitative methods review  Qualitative interviews  Focus groups  Naturalistic observation and participant observation  Video- and audio-taping  Anonymous open-ended survey  Document review (e.g. syllabi, exams)  Method choice

8. The Qualitative Day  Develop a plan for setting observation (in role)  Select settings (e.g. classrooms, mailroom, social events, training clinic staff meetings)  Plan focus  Plan for access and disclosure  Plan for observation and data recording  Plan for reporting

9. The Qualitative Day  Exercise on ethical reflection  Five minute individual writing on ethical concerns raised by the project  Class discussion

10. The Qualitative Day  Step back for big picture: history of paradigm conflict  Patton (2008)  Greene & Henry (2005)

11. The Qualitative Day  Alternative perspectives and choices  Little q: qualitative methods  BIG Q: Alternative paradigms  Paradigm choices and their implications  Positivism and post-positivism  Constructivism  Critical theory  Mixed methods and dialgogues

12. The Qualitative Day  Rationales for use  Always; sometimes; never  If sometimes: Situations of uncertainty, e.g. cross-cultural, unclear goals, process focus of stakeholder questions  Relevance to students’ own projects

13. The Qualitative Day: The Final Conversation  Judging quality  Lincoln & Guba, 1986  Trustworthiness  Authenticity  Caracelli & Riggin, 1994

14. The end! Reflect on it.