CHARACTERISTICS OF GOOD DESIGN Dr. Aidah Abu Elsoud Alkaissi Linköping University- Sweden An-Najah National University- Palestine 1

CHARACTERISTICS OF GOOD DESIGN In selecting a research design, researchers should be guided by one overarching consideration: whether the design does the best possible job of providing trustworthy answers to the research questions. Usually, a given research question can be addressed with a number of different designs, and researchers have flexibility in selecting one. Yet many designs are completely unsuitable for dealing with certain research problems. For example, a loosely structured design, such as those used in qualitative studies, would be inappropriate to address the question of whether non-nutritive sucking opportunities among premature infants facilitate early oral feedings. On the other hand, a tightly controlled study may unnecessarily restrict researchers interested in 2

CHARACTERISTICS OF GOOD DESIGN Understanding the processes by which nurses make diagnoses. There are many research questions of interest to nurses for which highly structured designs are unsuitable. TIP: Although techniques of research control are mechanisms for controlling bias, there are situations in which too much control can introduce bias. For example, if researchers tightly control the ways in which key study variables can manifest themselves, it is possible that the true nature of those variables will be obscured. When the key concepts are phenomena that are poorly understood or dimensions of which have not been clarified, then a design that allows some flexibility is better suited to the study aims. 3

CHARACTERISTICS OF GOOD DESIGN Cook and Campbell (1979), in their classic book on research design, describe four considerations that are important in evaluating research design for studies that focus on relationships among variables. The questions that must be addressed by researchers (and evaluated by research consumers) regarding research design are as follows: 1. What is the strength of the evidence that a relationship exists between two variables? 4

CHARACTERISTICS OF GOOD DESIGN 2. If a relationship exists, what is the strength of the evidence that the independent variable of interest (e.g., an intervention), rather than extraneous factors, caused the outcome? 3. If the relationship is plausibly causal, what are the theoretical constructs underlying the related variables? 4. If the relationship is plausibly causal, what is the strength of evidence that the relationship is generalizable across people, settings, and time? 5

CHARACTERISTICS OF GOOD DESIGN These questions, respectively, correspond to four aspects of a study’s validity: (1) statistical conclusion validity, (2) internal validity, (3) construct validity, and (4) external validity. In this section we discuss certain aspects of statistical conclusion validity, internal validity, and external validity, and factors that can undermine validity. 6

CHARACTERISTICS OF GOOD DESIGN Statistical Conclusion Validity The first criterion for establishing causality is demonstrating that there is, in fact, an empirical relationship between the independent and dependent variable. Statistical methods are used to determine if such a relationship exists. Design decisions can influence whether statistical tests will actually detect true relationships, and so researchers need to make decisions that protect against reaching false statistical conclusions. Although we cannot at this point in the text discuss all aspects of statistical conclusion validity, we can describe a few design issues that can be threats to making valid statistical inferences. 7

CHARACTERISTICS OF GOOD DESIGN Low Statistical Power Statistical power refers to the ability of the design to detect true relationships among variables. Adequate statistical power can be achieved in various ways, the most straightforward of which is to use a sufficiently large sample. When small samples are used, statistical power tends to be low, and the analyses may fail to show that the independent and dependent variables are related, even when they are. Another aspect of a powerful design concerns the construction or definition of the independent variable, and the counterfactual. Both statistically and substantively, results are clearer when differences between groups and treatments being compared are large. 8

CHARACTERISTICS OF GOOD DESIGN Researchers should usually aim to maximize group differences on the dependent variables by maximizing differences on the independent variable. In other words, the results are likely to be more clearcut if the groups are as different as possible. Conn et al (2001) offer excellent suggestions for strengthening the power and effectiveness of nursing intervention. 9

CHARACTERISTICS OF GOOD DESIGN Advice about strengthening group differences is more easily followed in experimental than in nonexperimental research. In experiments, investigators can devise treatment conditions that are distinct and as strong as time, money, ethics, and practicali permit. Even in nonexperimental research, however, there are frequently opportunities to operationalize independent variables in such a way that power to detect differences is enhanced. 10

CHARACTERISTICS OF GOOD DESIGN Inadequate Precision Quantitative researchers usually try to design a study to achieve the highest possible precision, which is achieved through accurate measuring tools, controls over extraneous variables, and powerful statistical methods. Precision can best be understood through a specific example. Suppose we were studying the effect of admission into a nursing home on depression by comparing elders who were or were not admitted. 11

CHARACTERISTICS OF GOOD DESIGN Depression varies from one elderly person to another for a various reasons. In the present study, we are interested in isolating—as precisely as possible—the portion of variation in depression attributable to nursing home admission. 12

CHARACTERISTICS OF GOOD DESIGN Mechanisms of research control that reduce variability attributable to extraneous factors can be built into the research design, thereby enhancing precision. In a quantitative study, the following ratio expresses what researchers wish to assess: This ratio, although greatly simplified here, captures the essence of many statistical tests. We want to make variability in the numerator (the upper half) as large as possible relative to variability in the denominator (the lower half), to evaluate clearly the relationship between nursing home admission and levels of depression. 13

CHARACTERISTICS OF GOOD DESIGN The smaller the variability in depression due to extraneous variables (e.g., age, prognosis), the easier it will be to detect differences in depression between elders who were or were not admitted to a nursing home. Designs that enable researchers to reduce variability caused by extraneous variables increase the precision of the research. As a purely hypothetical Variation in depression due to nursing home admission Variation in depression due to other factors (e.g., age, pain, medical prognosis, social support) 14

CHARACTERISTICS OF GOOD DESIGN illustration of why this is so, we will attach some numeric values* to the ratio as follows: If we can make the bottom number smaller, say by changing it from 4 to 2, then we will have a purer and more precise estimate of the effect of nursing home admission on depression, relative to other influences. All of the control mechanisms described in the previous section help to reduce variability caused by extraneous variables, and so should be considered in designing studies. 15

CHARACTERISTICS OF GOOD DESIGN We illustrate this by continuing our example. The total variability in levels of depression can be conceptualized by having three components: Total variability in depression Variability due to nursing home admission Variability due to age Variability due to other extraneous variables. 16

CHARACTERISTICS OF GOOD DESIGN This equation can be taken to mean that part of the reason why some elderly individuals are depressed and others are not is that some were admitted to a nursing home and others were not; some were older and some were younger; and other factors, such as level of pain, medical prognosis, availability of social supports, also had an effect on depression. One way to increase the precision in this study would be to control age, thereby removing the variability in depression that results from age differences. 17

CHARACTERISTICS OF GOOD DESIGN We could do this, for example, through homogeneity (i.e., by including in our sample only elderly people within a fairly narrow age range), by using age as a blocking variable, or by statistically controlling age. With any of these methods, the variability in depression due to age would be reduced or eliminated. As a result, the effect of nursing home admission on depression becomes greater, relative to the remaining extraneous variability. 18