1. University of Florida College of Medicine
‘Is it ‘random’ or ‘haphazard’? Demonstrating effects of nonrandom allocation by simulation
Penny S Reynolds, PhD
Anesthesiology
University of Florida College of Medicine
Gainesville FL USA

2. Randomization is the core of statistical inference - we know this BUT
Premise
Randomization is the core of statistical inference - we know this BUT

3. When random assignment, isn’t….
Problem
When random assignment, isn’t….
Investigators misunderstand
Analysts do not check

4. When random assignment, isn’t….
Trial by error
Observed distribution of baseline p-values is not consistent with randomization
When random assignment, isn’t….
Fraudulent data usually shows excessive high p-values ( fraudster want to indicate no differences). These indicate systematic differences & much more different than expected.
Reynolds and Garvan Military Medicine

5. Investigators conflate “haphazard”
What are they doing?
Investigators conflate “haphazard”
with “random”
Based on descriptions in published reports
Possible assignments:
‘Faux’ block (AAA…,BBB…,CCC…)
Alternating (ABC, ABC….)
The problem
Design basics
Remediation

6. Objectives
What are the consequences of non-random allocation for NHST in the presence of systematic trend ?
Simulation rationale

7. Toy population Typical of real data Simulation Models
Sample “population”: n = 100;
Linear trend Y range =
SAMPLE: N = 36; 3 treatments A B C; n = 12/group
TEST: H0: 1 = 2 = 3
Model
One-way ANOVA yi =  + ti + ei ei ~ N(0, 2)
With and without random effects, blocking
Toy population
Typical of real data

8. Assignment structures Model outputs
Restricted random (equal n/group)
RCBD
Alternating
Faux block
F-values
P-values
Cumulative type I error

9. Simulations: Run checks
Are simulations adequate?
Stability
Bias
Accuracy 5%
Cumulative error rate
replicate

10. Random allocation: test statistics
Analysis
Random effects?
F-value
Residual error P
Type III Fixed effects
Random Y
0.14
192.37
0.873 N
0.13
195.55
0.878
RCBD
block
0.61
6.60
0.552
(212.30)
Typical results:
Random
Assignment

11. Restricted random assignment: distributions
F distribution approximates theoretical;
P-values uniformly distributed
Type I error rate= 4.7%
Simulated F-values
Frequency
Simulated p-values
Frequency

12. Randomised complete block design
RCBD
Underlying distributions close to theoretical,
Increased efficiency
F distribution approximates theoretical;
P-values uniformly distributed
Type 1 error rate = 5.4%

13. Alternating ABC, ABC, ABC……..
F-values similar to randomized designs, BUT…….
Analysis
Random effects?
F-value
Residual error P
Type III Fixed effects
Random Y
0.14
192.37
0.873 N
0.13
195.55
0.878
RCBD
block
0.61
6.60
0.552
(212.30)
Alternating
0.27
0.685
0.763
0.11
194.71
0.896
Systematic (“faux”)
10.81
0.417
0.0003
155.57
18.80
<0.0001

14. Alternating: Distributions
Distribution collapse
Small F;
High p
Type 1 error rate = 100%
Simulated F-values

15. Faux blocks: AAA…, BBB…, CCC…
Analysis
Random effects?
F-value
Residual error P
Type III Fixed effects
Random Y
0.14
192.37
0.873 N
0.13
195.55
0.878
RCBD
block
0.61
6.60
0.552
(212.30)
Alternating
0.27
0.685
0.763
0.11
194.71
0.896
Systematic (“faux”)
10.81
0.417
0.0003
155.57
18.80
<0.0001
Systematic trend contributes to false positives

16. Faux blocks: Distributions
Distribution collapse:
high F,
small p
Type 1 error rate = 100%
126.36
P-values

17. No meaningful test can be based on non-random ‘designs’
The null distribution is the appropriate reference distribution only if assignment is actually randomized. If not randomized, inferential statistics CANNOT provide valid probability statements about treatment effects.
CONCLUSIONS:
Statistical

18. How were the data collected?
Before analysis and NHST we need to know how data are actually acquired Do data meet assumptions for subsequent analyses?
CONCLUSIONS:
For consultants