1. Research design Dr Brenda Louw 2010

2. Objectives  To demonstrate k nowledge and understanding of research designs  To differentiate between quantitative, qualitative and mixed-methods research design  To differentiate between group and single subject quantitative designs  To identify the design in a research article  To evaluate the match between the research question and the research design  To select an appropriate research design for a planned research project

3. Overview Introduction Types of research design Quantitative research design Group research design Single subject research design Qualitative research design Mixed methods research design

4. Readings Schiavetti et al.,2011 Chapter 4

5. Introduction In planning research the investigator requires an overall research strategy General plan of attack Then a RESEARCH DESIGN can be selected Tactics used to carry out the strategy Devise ways in which to implement the design Plan for selecting and measuring IV and DV to answer research questions about their relationships Dual purpose Answer questions posed Control for variance (alternative explanations, factors that hinder meaningful interpretation)

6. Research Design: two purposes In experimental research: to manipulate the IV in order to explore the effects of this manipulation on the DV; to arrange the experiment in such a way that extraneous variables are controlled and cannot have a confounding effect on the DV. In descriptive research: to select the variables for observation to explore the dimensions or differences or relationships found in the natural phenomena; to make those observations in a systematic and unobstructive fashion, so that the behavior found in the criterion variables are not confounded by the extraneous variables.

7. Types Research design Quantitative research design Group research design Between-subjects designs Within-subjects designs Mixed designs Single subject research design ( Dr Williams) Qualitative research design Mixed Methods research design

8. QUANTITATIVE RESEARCH DESIGN Group designs: one or more groups of participants are exposed to one or more levels of the IV; the average performance of the group of participants on the DV is examined to determine the relationship between the IV and DV. Single-subject design: focus on the individual behavior of participants; more than one person may be evaluated but the data of each person is evaluated individually without performing a group average

9. Both approaches apply the components of the scientific method to their approach to research - the difference between group and single-subject designs lies in the manner in which the principles of the scientific method are put into operation in designing studies

10. Group designs Power lies in the statistical significance Control and experimental groups Random selection of Ss Employs statistical tests Variability managed through statistical analysis

11. Group designs 3 categories: Between-subjects design Within-subjects design Mixed design Simple group designs Complex group designs

12. Group designs WHY Group? Research that gathers quantitative data from Ss to determine cause-effect relationship/ association between variables employs either 3 types: Between-subjects design Within-subjects design Mixed design

13. Group designs i) Between-Subject designs Experimental research : Performances of separate groups of Ss measured and compared Different groups of Ss are exposed to different IVs t he IV is applied to one group (exp) but not to the other group (control) IV or experimental treatment applied to experimental group but not applied to another group=control group

14. Between-Subject designs Equivalence of experimental and control groups important in the evaluation of the design 2 groups of Ss exposed to 2 levels of IV but are different ito e.g. age, gender, experience they may perform differently on the DV due to their characteristics rather than because they have been exposed to different levels of the IV

15. Between-Subject designs Ensure group equivalence between experimental and control groups in all respects except for the IV in the 2 groups The major threat to internal validity is subject selection differences may be Ss selection differences rather than the IV

16. Between-Subject designs 2 ways to equate experimental and control groups: randomization matching

17. Between-Subject designs Subject Randomization Assign Ss in experimental and control group on a random basis Equal probability of being assigned to either group Subject matching Match members of 2 groups on all extraneous variables relevant to the experiment

18. Between-Subject Designs Descriptive research : Different groups of Ss responses compared with each other with regard to their performance on some criterion variable Groups differ re classification variable

19. Between-Subject Designs Select Ss who fall distinctly into the different categories of the classification variable but who are otherwise equivalent with regard to extraneous variables Definition of criteria for selecting Ss 1 st step in design classifications must be constructed that are mutually exclusive Ss fall into only one of two groups

20. Between-Subject Designs 2 nd step in design of between-subjects descriptive research is to equate subjects on extraneous variables Random assignment not possible Overall and pair matching

21. Between-Subject Designs Experimental Compare groups exposed to different treatments or levels of the IV Descriptive Compare performances of Subjects in different classifications

22. Group designs ii) Within-Subjects Designs The performance of the same subjects is compared in different conditions. In experimental research, the subjects are exposed to all treatment or levels of the IV basic concern with these designs is that all conditions should be equivalent except for the application of the IV therefore, it is important to control for sequencing or order effect

23. Within-Subjects Designs There are 2 ways to control for the sequencing effect: randomization presentation of the experimental treatment conditions (IV) is randomly sequenced counterbalancing arrange all possible sequences of treatments (IV) and then randomly assign subjects to each sequence

24. Within-Subjects Designs In descriptive research, longitudinal studies would be a within-subjects design Same Subjects as the age and mature Rate of development as time passes

25. Group Research Designs: iii) Mixed designs Combination of between-subjects and within- subjects designs More than 1 IV considered

27. Group Designs Simple Group Designs one IV with 2 levels 2 levels can be independent groups (Exp + Cont) 2 levels can be repeated measurements (pre/post) Complex Group Designs one or more IVs factorial designs more than 2 levels on the IV more than 1 DV multivariate designs

28. Need to keep 2 things in mind simultaneously: Independent Variable # levels 2 levels (bi-valent) --> SIMPLE > 2 levels (multi-valent) --> COMPLEX # variables 1 IV (simple OR complex group design) 2 or more IVs --> COMPLEX Groups Independent Groups Repeated Measures

29. These 2 things can be mix-matched to come up with different design combinations. Ex: 2 IVs with 2 levels each in an independent group design (2 x 2 independent group design)

30. SimpleComplex Independent Groups Repeated Measures # Levels/# IVs# Levels/# IVs 1 IV> 1 IV 2 levels > 2 levels (bivalent) (multivalent)

31. Simple Group Designs Independent Group Designs random selection designs random assignment designs matched group designs natural group designs Repeated Measurement Designs Simple Correlational Designs

32. Simple Group Designs Involve 1 IV with 2 levels and 1 DV the levels of the IV can be independent groups or repeated measurements

33. 4 Types of Independent Simple Group Designs random selection designs random assignment designs matched group designs natural group designs

34. i)Random Selection Designs 2 groups are randomly selected from the same population one group receives one level of the IV and the other group receives the other level the effect of varying the IV is indicated by the difference between groups on the DV this simple design doesn’t provide much control of subject variables such as age, gender, and education which researchers generally prefer to control

35. ii) Random Assignment Designs When only a small population of subjects is available, they can be randomly assigned to one group or the other. This is the only difference from the random selection designs, that is, subjects are selected from a smaller population subject variables are controlled by allowing them to vary randomly across both groups

36. Random Selection & Assignment Selection refers to the process by which a sample is recruited from a population. A random selection of observations from the population will generally yield a sample that is representative of the entire population. Self-selection and other non-random recruitments will constrain the population and therefore, the ability to generalize the study findings. Assignment refers to the process by which the sample is further divided into experimental and control groups. A random assignment of the sample will generally yield groups that are similar to each other and the difference among them will be attributable to the experimental conditions. Non-random assignment procedures limit a study’s ability to control for non-experimental characteristics.

37. Random Selection & Assignment Random Selection from a population makes use of "chance" to obtain a sample that is representative of the population on key variables. Random Assignment is done before the start of an experiment to obtain control and treatment groups that are equivalent; e.g., everyone has equal chance of being in tx group. * O x * * x * x O * O *x O * * Population Sample

38. Random Selection & Assignment It is possible to have only one of these but not the other in a study. You take the first 100 clients on the list (no random selection) and randomly assign this nonrandom sample to treatment versus control. You randomly select 100 from the list of 1000 and then nonrandomly assign them to treatment or control.

39. Random Selection & Assignment Random selection is most related to the external validity (or generalizability): a sample is random and should represent the larger group from which the participants are drawn. We ask the question: with what other groups could we reasonably expect to get the same results if we used the same treatment? Random assignment is related to internal validity: assignment is random to assure that our treatment groups are similar to each other (i.e., equivalent) prior to the treatment. We ask the question: does it seem reasonable to assume that the treatment has really produced the measured effect?

40. iii) Matched Group Designs One or more variables that may affect the DV is held constant between groups by matching the groups on those variables Thus the problem of subject variability that was a problem in random selection designs is overcome with this design

41. Matched Group Designs there are 2 types of matched group designs groups can be matched on the DV (e.g., vocabulary skills, test scores, etc.) groups can be matched on variables that might affect the DV (e.g., age, gender, education) this design is more useful to CD researchers because of the small groups that are often available to researchers

42. iv) Natural Group Designs 2 groups selected from two different populations In this design, the IV is a difference between the groups created by nature that exists prior to the selection of the groups. It is the effect of this IV (i.e., difference between the groups) that is studied

43. Repeated Measurement Designs This design has a single group of subjects in which the two levels of the IV are varied within the same group of subjects this design is used when there are not enough subjects available for two independent groups or when it is more efficient to carry out the experimental procedures within one group

44. Repeated Measurement Designs The DV is assessed twice in a single group of subjects the difference between the two measurements demonstrates the effect of the IV a problem with this design is the practice effect of repeating a measurement. Another problem is the order effect of measurements administered to subjects. To control for this, the researcher should use counterbalancing of the order of measurements to subjects.

45. Number of Subjects Needed for Simple Group Designs 20 (10 per group for independent groups OR 20 for repeated measures) In CDIS, the absolute minimum would be 10 subjects (5 per group for independent groups OR 10 for repeated measures)

46. Complex Group Designs Complex designs extend the simple group designs. More than 1 IV may be studied; more than 2 levels of the IV may be studied; and more than 1 DV may be examined. In addition, independent group designs and repeated measurement designs may be combined.

47. Designs with more than 2 levels of the IV Independent Group Designs: more than two levels of the IV is examined; more information obtained. For example, comparing the effects of 3 levels of training (method a, method b, control). Repeated Measurement Designs: assessing more than two things. EX: IV - type of hearing aid (HA); Levels: HA-1, HA-2, HA-3; same subjects are tested on all three levels (or HAs). Order effects are important; must counterbalance the order of presentation of tests, assessments, or measurements. Six possible orders for 3 levels (1 2 3; 1 3 2; 2 1 3; 2 3 1; 3 1 2; 3 2 1); 24 orders for 4 levels; 120 for 5 levels; use of incomplete counterbalancing.

48. Designs with more than 1 IV (Factorial Designs) Designs that vary two or more IVs at same time can provide detailed information related to the complexity of the processes and disorders of communication. Factorial designs can involve independent groups, repeated measures, or both (mixed factorial designs). The more complex the design, the greater the number of experimental conditions (or cells) in the factorial design: two IVs with 2 levels each is a 2 x 2 (4 cells); three IVs with 2 levels each is a 2 x 2 x 2 (8 cells). With factorial designs, you can determine if there are main effects of each of the IVs as well as an interaction effect between the IVs.

49. Simple Factorial Designs The simplest factorial design has 2 IVs with 2 levels each. The 2 IVs can: both be independent groups; both be repeated measures; one independent group and one repeated measure (mixed factorial design). 2 x 2 Independent Group Design Two groups that differ with respect to 2 different IVs, e.g., normal vs. disordered; AND male vs. female. NormalDisordered MaleGroup1Group3 Sex FemaleGroup2Group4

50. |N | N no interaction |D |_D__________ MF |D | N interaction |N |_D__________ MF

51. 2 x 2 Repeated Measurement Design One group that received two different measurements, e.g., tested HA 1 vs. HA 2 in noisy vs. quiet conditions; must control for order effects. Type of HA HA1HA2 NoisyGroup1 Condition QuietGroup1

52. |2 | 2 no interaction |1 |_1__________ NQ |1 | 2 interaction |2 |_1__________ NQ

53. 2 x 2 Mixed Design Two groups that receive some assessment, e.g., normal vs. disordered (independent group design) AND pretest vs. posttest (repeated measure design). Group NormalDisordered PretestGroup1Group2 PosttestGroup1Group2

54. 2 x 2 Mixed Design Two groups that receive two different treatments AND have different disorders (mild vs. severe). Treatment TxATxB MildGroup1 SevereGroup2

55. Complex Factorial Designs Factorial designs can be made more complex by increasing the number of IVs, the number of levels of the IVs, or both. There can be 3 or more IVs and 3 or more levels of each IV. These designs are interpreted the same way as simple factorial designs, but there are many more possible outcomes. Therefore, the complex factorial design can provide more information about the complex interactions.

56. Example of a 2 x 2 x 2 factorial design Evaluation of the effects of linguistic complexity on stuttering; previous studies suggest importance of gender and the level of education. Independent VariablesLevels Linguistic complexitySimple vs. Complex Repeated Measurement GenderMale vs. FemaleIndependent Groups Education High School vs. University Independent Groups There are 8 cells (4 groups times two repeated measurements). The limitations of complex factorial designs are: (1) the number of subjects and (2) the number of experimental conditions required by the design.

57. Number of Subjects Needed for Complex Group Designs 5-10 subjects per independent group or repeated measurement cell. Thus, a minimum of 20-40 Ss would be needed for the 4 cells of a 2 x 2 factorial design and a minimum of 300-600 Ss for the 60 cells of a 3 x 4 x 5 factorial design.

58. Advantages of Group Designs Isolate the effects of the IVs by systematically varying the levels of the IVs to determine the effects on the DVs the IVs can be independent groups, repeated measures, or both control the effects of other variables by allowing them to vary randomly in random selection designs, by holding them constant in matched group designs, or by systematically varying them in factorial designs

59. Advantages of Group Designs Provide information about interaction effects of IVs on the DV generalizability of findings demonstrate causal relationships

60. Disadvantages of Group Designs Difficulty obtaining large numbers of subjects thus, groups may be small and the number and levels of the IVs may be restricted such restrictions limit the interpretations of results obtained with group designs and decrease the knowledge that can be obtained with these findings

61. Disadvantages of Group Designs Group averages may not adequately represent the characteristics of individuals quantified measures of the DV may not provide enough information may not apply to natural settings

62. SINGLE SUBJECT DESIGNS Guest lecturer Dr Williams

63. QUALITATIVE RESEARCH DESIGN Qualitative research primarily exploratory Focuses on generating hypotheses, identifying key variables Research design antithetical ! Different qualitative research strategies but no set of defined research designs Qualitative designs ever developing plan

64. Continued Design emerges as study progresses and ongoing refinement of Q and methods takes place 3 characteristics features of qualitative research designs: Design flexibility-starts with ideas, open to change, adjusts as data is collected Purposeful sampling-criterion-based selection Naturalistic inquiry- participatory designs

65. MIXED METHODS RESEARCH DESIGN Combination of quantitative and qualitative designs Employ,multiple approaches in all stages of the research –from problem identification and research questions, to data collection and analysis Complex but allows researchers multiple perspectives

66. CONCLUSION

67. Conclusion Evaluation of research design Internal validity: concerns the degree to which the design meets two purposes, i.e., providing answers to research questions and controlling variability. External validity: concerns the degree to which generalizations can be made outside of the scope of the study. Class 6