1. Probability Sampling uses random selection
N = number of cases in sampling frame
n = number of cases in the sample
NCn = number of combinations of n from N
f = n/N = sampling fraction

2. Variations Simple random sampling Stratified random sampling
based on random number generation
Stratified random sampling
divide pop into homogenous subgroups, then simple random sample w/in
Systematic random sampling
select every kth individual (k = N/n)
Cluster (area) random sampling
randomly select clusters, sample all units w/in cluster
Multistage sampling
combination of methods

3. Nonprobability sampling
accidental, haphazard, convenience sampling ...
may or may not represent the population well

4. Measurement
... topics in measurement that we don’t have time to cover ...

5. Research Design Elements: Samples/Groups Measures Treatments/Programs
Methods of Assignment
Time

6. Internal validity
the approximate truth about inferences regarding cause-effect (causal) relationships
can observed changes be attributed to the program or intervention and NOT to other possible causes (alternative explanations)?

7. Establishing a Cause-Effect Relationship
Temporal precedence
Covariation of cause and effect
if x then y; if not x then not y
if more x then more y; if less x then less y
No plausible alternative explanations

8. Single Group Example Single group designs:
Administer treatment -> measure outcome
X -> O
assumes baseline of “0”
Measure baseline -> treat -> measure outcome
0 X -> O
measures change over baseline

9. Single Group Threats History threat Maturation threat Testing threat
a historical event occurs to cause the outcome
Maturation threat
maturation of individual causes the outcome
Testing threat
act of taking the pretest affects the outcome
Instrumentation threat
difference in test from pretest to posttest affects the outcome
Mortality threat
do “drop-outs” occur differentially or randomly across the sample?
Regression threat
statistical phenomenon, nonrandom sample from population and two imperfectly correlated measures

10. Addressing these threats
control group + treatment group
both control and treatment groups would experience same history and maturation threats, have same testing and instrumentation issues, similar rates of mortality and regression to the mean

11. Multiple-group design
at least two groups
typically:
before-after measurement
treatment group + control group
treatment A group + treatment B group

12. Multiple-Group Threats
internal validity issue:
degree to which groups are comparable before the study
“selection bias” or “selection threat”

13. Multiple-Group Threats
Selection-History Threat
an event occurs between pretest and posttest that groups experience differently
Selection-Maturation Threat
results from differential rates of normal growth between pretest and posttest for the groups
Selection-Testing Threat
effect of taking pretest differentially affects posttest outcome of groups
Selection-Instrumentation Threat
test changes differently for the two groups
Selection-Mortality Threat
differential nonrandom dropout between pretest and posttest
Selection-Regression Threat
different rates of regression to the mean in the two groups (if one is more extreme on the pretest than the other)

14. Social Interaction Threats
Problem:
social pressures in research context can lead to posttest differences that are not directly caused by the treatment
Solution:
isolate the groups
Problem: in many research contexts, hard to randomly assign and then isolate

15. Types of Social Interaction Threats
Diffusion or Imitation of Treatment
control group learns about/imitates experience of treatment group, decreasing difference in measured effect
Compensatory Rivalry
control group tries to compete w/treatment group, works harder, decreasing difference in measured effect
Resentful Demoralization
control group discouraged or angry, exaggerates measured effect
Compensatory Equalization of Treatment
control group compensated in other ways, decreasing measured effect

16. Intro to Design/ Design Notation
Observations or Measures
Treatments or Programs
Groups
Assignment to Group
Time

17. Observations/Measure
Notation: ‘O’
Examples:
Body weight
Time to complete
Number of correct response
Multiple measures: O1, O2, …

18. Treatments or Programs
Notation: ‘X’
Use of medication
Use of visualization
Use of audio feedback
Etc.
Sometimes see X+, X-

19. Groups
Each group is assigned a line in the design notation

20. Assignment to Group R = random N = non-equivalent groups
C = assignment by cutoff

21. Time
Moves from left to right in diagram

22. Types of experiments True experiment – random assignment to groups
Quasi experiment – no random assignment, but has a control group or multiple measures
Non-experiment – no random assignment, no control, no multiple measures

23. Design Notation ExampleR O1 X
O1,2
Pretest-posttest treatment versus
comparison group
randomized experimental design

24. Design Notation Example
Pretest-posttest
Non-Equivalent Groups
Quasi-experiment

25. Design Notation Example
Posttest Only
Non-experiment

26. Goals of design .. Goal:to be able to show causality
First step: internal validity:
If x, then y
AND
If not X, then not Y

27. Two-group Designs Two-group, posttest only, randomized experiment R X
Compare by testing for differences between means of groups, using t-test or one-way Analysis of Variance(ANOVA)
Note: 2 groups, post-only measure, two distributions each with mean and variance, statistical (non-chance) difference between groups

28. To analyze …
What do we mean by a difference?

29. Possible Outcomes:

30. Three ways to estimate effect
Independent t-test
One-way Analysis of Variance (ANOVA)
Regression Analysis (most general)
equivalent

31. The t-test
appropriate for posttest-only two-group randomized experimental design
See also: paired student t-test for other situations.

32. Measuring Differences …

33. Computing the t-value

34. Computing standard deviation
standard deviation is the square root of the sum of the squared deviations from the mean divided by the number of scores minus one
variance is the square of the standard deviation

35. ANOVA
One-way analysis of variance

36. ANOVA
Analysis of variance – tests hypotheses about differences between two or more means
Could do pairwise comparison using t-tests, but can lead to true hypothesis being rejected (Type I error) (higher probability than with ANOVA)

37. Between-subjects design
Example:
Effect of intensity of background noise on reading comprehension
Group 1: 30 minutes reading, no background noise
Group 2: 30 minutes reading, moderate level of noise
Group 3: 30 minutes reading, loud background noise

38. Experimental Design One factor (noise), three levels(a=3)
Null hypothesis: 1 = 2 = 3
Noise
None
Moderate
High R O

39. Notation If all sample sizes same, use n, and total N = a * n
Else N = n1 + n2 + n3

40. Assumptions Normal distributions Homogeneity of variance
Variance is equal in each of the populations
Random, independent sampling
Still works well when assumptions not quite true(“robust” to violations)

41. ANOVA Compares two estimates of variance If null hypothesis
MSE – Mean Square Error, variances within samples
MSB – Mean Square Between, variance of the sample means
If null hypothesis
is true, then MSE approx = MSB, since both are estimates of same quantity
Is false, the MSB sufficiently > MSE

42. MSE

43. MSB Use sample means to calculate sampling distribution of the mean,
= 1

44. MSB Sampling distribution of the mean * n
In example, MSB = (n)(sampling dist) = (4) (1) = 4

45. Is it significant? Depends on ratio of MSB to MSE F = MSB/MSE
Probability value computed based on F value, F value has sampling distribution based on degrees of freedom numerator (a-1) and degrees of freedom denominator (N-a)
Lookup up F-value in table, find p value
For one degree of freedom, F == t^2

46. Factorial Between-Subjects ANOVA, Two factors
Three significance tests
Main factor 1
Main factor 2
interaction

47. Example Experiment Two factors (dosage, task)
3 levels of dosage (0, 100, 200 mg)
2 levels of task (simple, complex)
2x3 factorial design, 8 subjects/group

48. Summary table Sources of variation: Task Dosage Interaction Error
SOURCE df Sum of Squares Mean Square F p
Task
Dosage TD
ERROR
TOTAL
Sources of variation:
Task
Dosage
Interaction
Error

49. Results Sum of squares (as before)
Mean Squares = (sum of squares) / degrees of freedom
F ratios = mean square effect / mean square error
P value : Given F value and degrees of freedom, look up p value

50. Results - example
Mean time to complete task was higher for complex task than for simple
Effect of dosage not significant
Interaction exists between dosage and task: increase in dosage decreases performance on complex while increasing performance on simple

51. Results

52. Regression Analysis
Equivalent to t-test and ANOVA for post-test only two group factorial design

53. Regression Analysis
Solve overdetermined system of equations for β0 and β1, while minimizing sum of e-terms

54. Regression Analysis

55. ANOVA Compares differences within group to differences between groups
For 2 populations, 1 treatment, same as t-test
Statistic used is F value, same as square of t-value from t-test

56. Other Experimental Designs
Signal enhancers
Factorial designs
Noise reducers
Covariance designs
Blocking designs

57. Factorial Designs

58. Factorial Design Factor – major independent variable
Setting, time_on_task
Level – subdivision of a factor
Setting= in_class, pull-out
Time_on_task = 1 hour, 4 hours

59. Factorial Design Design notation as shown
2x2 factorial design (2 levels of one factor X 2 levels of second factor)

60. Outcomes of Factorial Design Experiments
Null case
Main effect
Interaction Effect

61. The Null Case

62. The Null Case

63. Main Effect - Time

64. Main Effect - Setting

65. Main Effect - Both

66. Interaction effects

67. Interaction Effects