1. Research Methods Observations Interviews Case Studies Surveys Quasi Experiments

2. Classes of Research Methods  Experimental  Non-Experimental Observations Interviews Case studies Surveys Quasi-Experiments

3. Intervention Target system Outcomes

4. Approach Comparison Adapted from Yin (1994)

5. Experiments

6.  Most common method of evaluation  Primary components Desired intervention or change Inputs (what you vary or keep the same) Outcomes (what you are looking for)  Vary one or more inputs and look for differences in the outcomes.  Randomize

7. Experiments Strengths  Can be used to show cause and effect  Quantitative techniques can be used to show strength of relationships  Accepted across a wide- range of disciplines Limitations  In the “real world” in may be difficult to have random assignment  May not be able to create realistic conditions in a controlled setting

8. Observations

9.  Used to capture the ways in which people act and interact within a particular environment/setting  Observations may be either qualitative or quantitative  Researcher will either use detailed field notes, audio or video recording to capture data  Focus may be on general aspects of behavior/phenomenon for qualitative; for quantitative, typically will focus on a particular aspect of behavior that can be quntified

10. Strengths  Extremely flexible – researcher can shift focus as new data come to light  Good method to study behaviors and other aspects of a system that may be difficult to quantify  Enables researcher to study the richness and complexity of human behavior  Data may be taken at multiple points in time

11. Limitations  Behavior to be studied must be defined in a precise, concrete manner to be recognizable  Typically takes multiple researchers  Researcher’s presence may impact participants’ behaviors  Requires meticulous attention to detail in planning, data collection, and data analysis  Results are limited to the system studied

12. Reliability How reproducible is the data? How consistent is the data?  Data collection protocol must exist (but may be modified as research progresses)  Data analysis training to assure consistency between researchers  Data analysis reliability assessment if multiple coders are used

13. Validity How well does the data collected address the characteristics of interest?  Face validity  Content validity  Criterion validity  Construct validity

14. Interviews

15.  Can produce a great deal of useful information  Can be structured, open-ended, or semi- structured  Can be used for individuals or groups of individuals (focus groups)  IRB – confidentiality and informed consent must be planned well in advance

16. Interview tips  Make sure interviewees are representative  Find a suitable location  Take a few minutes to establish rapport  Don’t put words in people’s mouths  Record responses verbatim  Keep your reactions to yourself  Use contingency questions to avoid irrelevant questions

17. Choosing Interviewees  Interviewees must be competent to answer  Interviewees must be willing to answer  Questions should be relevant to the Interviewees

18. Strengths  Rich and complete information  Question complexity can be greater than survey  Flexible (particularly if open-ended)

19. Limitations  Can produce information that is not related to research topic and/or not comparable to other data collected  More expensive than written surveys  Time consuming  Sensitive questions may not be answered honestly

20. Case studies

21. Case Studies  An evaluation method that “investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident” (Yin 1994, p. 13)

22. Case studies  Used to study an individual, program, event, or organization in depth for a pre- defined period of time  Researcher must know what he/she is looking for  Use multiple sources of data, e.g. interviews, direct observation, document analysis, or surveys

23. Case Studies Strengths  Can be done after the fact  Valuable technique to highlight learnings from a project  Data collection methods are relatively straight-forward Limitations  Difficult to generalize the results to other parts of the organization or to other organizations.  Will not show cause and effect relationships  Time consuming

24. Quality Digest Case Study Example Taking SPC to the production line  Target System: Organizations that have front line workers using SPC tools  Types of data: Direct observation, interviews, and documents  Findings: Lessons learned, road blocks, and best practices were highlighted

25. Surveys

26. Why use surveys?  A method for collecting data to describe some specific characteristics of individuals or groups of people.  A method for measuring some specific attitudes, perceptions, and beliefs of individuals or groups of people.

27. Survey design issues  Sample selection  Sample size  Data collection method  Questionnaire format  Question construction

28. Surveys  Understand how people view a particular topic without asking every person  Good method to quantify otherwise qualitative information  Data may be taken at multiple points in time  Use questionnaires or structured interviews to collect information Adapted from Creswell (1994)

29. New Program Evaluation Example SWE e-mail mentoring program  Target System: Program participants  Intervention: Mentors/mentees exchange e-mails about careers in engineering  Outcomes: Are mentees more likely to consider engineering careers than before? Do mentors and mentees feel more positive about themselves?

30. Surveys Strengths  Can be used to create quantitative measures of “softer” types of data.  Can be done at many points in time if needed  Methods for evaluating the data exist Limitations  Results are limited to the overall group represented in the survey sample  Does not give you a way to follow up if results are inconclusive.

31. Quasi-Experiments

32.  Used in social science, education, policy, and management systems  Use when randomization of target system and or intervention is not possible  Multiple-designs exist  Primary components Study group Comparison group Intervention

33. Quasi-Experimental Approach

34. Nonequivalent Control Group Example O1XO2O1O2O1XO2O1O2

35. Autonomous Team Example Implementing teams in a mineral plant  Study group: Autonomous teams in a startup plant  Comparison group: Existing shift workers in the original plant  Intervention: Creation of teams  Outcomes: Job satisfaction, trust, and productivity

36. “Ideal Outcomes”

37. Trends in Opposite Direction

38. Trends with Differing Growth Rates

39. Quasi-Experiments Strengths  Can be used in the “real world”  Does not require changes to existing organization  Designs and techniques for evaluation exist Limitations  Can not be used to show cause and effect  May not be able to generalize the results  Results can potentially be explained by other factors