1 Evaluating Research This lecture ties into chapter 17 of Terre Blanche We know the structure of research Understand designs We know the requirements of “good” research Now we can evalute a study Is it good? Can we believe its conclusions?

2 Validity in designs If a design is not valid, then the conclusions drawn are not supported Like not doing research at all Validity is evaluted before the design is run, and problems solved/worked around Validity of designs come in 2 parts: Internal validity (can the design sustain the conclusions?) External validity (can the conclusions be generalized to the population effectively?)

3 Internal validity Each design is only capable of supporting certain types of conclusions Eg. Only experiments can support conclusions about causality Says nothing about if the results can be applied to the real world Generally, the more controlled the situation, the higher the internal validity

4 External Validity Can the findings of the study be generalized? Do they speak only of our sample, or of a wider group? Says nothing about the truth of the result being generalized Generally, bigger samples with valid measures lead to better external validity

5 Examining the validity of designs Each of the three major design types has different internal/external validity requirements We can examine the aims of each of these to determine how much weight should be given to internal and external validity

6 Validity of descriptive designs Aim: accurately describe the world The central purpose is to describe the population External validity issue is central Internal validity is not irrelevant Measurement instruments Poor scales will paint the wrong picture

7 Validity of descriptive designs (2) Two important questions to ask How good was the sample? Were correct sampling techniques used? Does the sample represent the population Is the target population defined? How good was the measurement? Were the variables selected representative of the behaviours? Were reliable/valid scales used?

8 Validity of relational designs Aim: discover relationaships between variables Observation of at least 2 variables Generally, only external validity is an issue Want the relationships to generalized Main internal validity problem is causality Causality cannot be sustained by a relational design

9 Internal validity of relational designs Few problems, if you remember that it cannot show causation Why? Mediator/Moderator variables (change the strength of a relationship without taking part) Problem of direction What is causing what?

10 External Validity of Relational Designs Focuses on naturally occuring relationships Thus external validity is ver important Populations must be carefully defined Samples selected to represent them well Measures must be both reliable & valid Unreliable / invalid measures can lead to no picture/false picture

11 Validity of Experimental designs Aim: Establish that a relationship is causal External validity is fairly important Especially to apply findings (eg. cures) Most important: Internal validity Only by sticking to the strict design of the experiment is it possible to show causality

12 Control in experiments To show that the IV causes the DV, the only difference between control and experimental group must be the IV manipulation Any other differences between the groups is a confound Something else that may have caused the DV

13 Control in experiments The best way to ensure that there are no differences between experimental and control groups is to assign subject carefully Eg. have equal number of males and females in both groups How do we know that the groups are evenly split on all charactertistics? Intelligence, gender, personality type, optimism….

14 Random Assignment It is impossible to manually create even groups We do not know all the characteristics that may confound the experiment Answer: Random assignment Ensures that evenly balanced groups will result, but only if the sample size is fairly large (n > 30)

15 Campbell’s Scheme Detecting what threatens the validity of an experiment is difficult Donald Campbell devised a system to make the system easier Designed for experiments, but can be applied to all designs Gives a list of possible threats to internal and external validity Check the design to these criteria

16 Threats to internal validity 1. Co-varying events Another, unseen variable might be causing the effect we are seeing 2. Maturation Changes over time can be caused by a natural learning process 3. Reactivity (testing effect) People realize that they are being studied, and respond they way they think is appropriate

17 Internal validity threats 4. Instrument decay Instruments with low reliability lead to inaccurate findings/missing phenomena 5. Regression to the mean Studying extreme scores can lead to inflated differences, which would not occur in moderate scorers 6. Subject mortality If subjects drop out, it creates a bias to those who didn’t

18 External validity threats 1. Subject selection Selecting a sample which does not represent the population well will prevent generalization 2. Operationalization of the variables We take a variable (wide scope) and operationalize it (narrow scope) – will we find the same results with a different operationalization of the same variable?