1. Design of Experiments and Data Analysis
Graduate Seminar
Fall 2006
Hal Carter
These slides available at
Bibliography
1. Julian Faraway, “Practical Regression and Anova using R,” July Avail at cran.us.r-project.org/other-docs.html
2. “The R Project for Statistical Computing,” Available at Software available for Linux, Windows, and OS-X.
3. Raj Jain, “The Art of Computer Systems Performance Analysis,” Wiley, 1991.
4. “An Introduction to R,” Available at

2. Agenda Analyze and Display Data Design Your Experiments
Simple Statistical Analysis
Comparing Results
Determining O(n)
Design Your Experiments
2K Designs Including Replications
Full Factor Designs

3. A System
Factors
System
System
inputs
System
outputs
Responses

4. Experimental Research
Define
System
Define system outputs first
Then define system inputs
Finally, define behavior (i.e., transfer function)
Identify
Factors
and Levels
Identify system parameters that vary (many)
Reduce parameters to important factors (few)
Identify values (i.e., levels) for each factor
Identify
Response(s)
Identify time or space effects of interest
Design
Experiments
Identify factor-level experiments

5. Create and Execute System; Analyze Data
Define
Workload
Workloads are inputs that are applied to system
Workload can be a factor (but often isn't)
Create
System
Create system so it can be executed
Real prototype
Simulation model
Empirical equations
Execute
System
Execute system for each factor-level binding
Collect and archive response data
Analyze &
Display
Data
Analyze data according to experiment design
Evaluate raw and analyzed data for errors
Display raw and analyzed data to draw conclusions

6. Some Examples Epitaxial growth Analog Simulation
New method using non-linear temp profile
What is the system?
Responses
Total time
Quality of layer
Total energy required
Maximum layer thickness
Factors
Temperature profile
Oxygen density
Initial temperature
Ambient temperature
Analog Simulation
Which of three solvers is best?
What is the system?
Responses
Fastest simulation time
Most accurate result
Most robust to types of circuits being simulated
Factors
Solver
Type of circuit model
Matrix data structure

7. SIMPLE MODELS OF DATA Evaluation of a new wireless network protocol.
System: wireless network with new protocol
Workload:
10 messages applied at single source
Each message identical configuration
Experiment output:
Roundtrip latency per message (ms)
Data file “latency.dat”
Mean: 19.6 ms
Variance: ms2
Std Dev: 3.27 ms
Latency 22 23 19 18 15 20 26 17
% R
> data=read.table("latency.dat",header=T)
> data
Latency
> attach(data)
> mean(Latency)
[1] 19.6
> var(Latency)
[1]
> sd(Latency)
[1]
> Index=c(1:10)
> plot(Index,Latency,pch=19,cex.lab=1.5)

8. Verify Model Preconditions
Check normal distribution
`Use quantile-quantile plot
Pattern adheres consistently along
ideal quantile-quantile line
Check randomness
Use plot of residuals around mean
Residuals appear random
> # Plot residuals to assess randomness
> Residuals=Latency-mean(Latency)
> plot(Index,Residuals,pch=19,cex.lab=1.5)
> abline(0,0)
> # Plot quantile-quantile plot to assess if residuals
> # normally distributed
> qqnorm(Latency, pch=19,cex.lab=1.5)
> qqline(Latency)

9. For the latency data, m = 10, a = 0.05:
Confidence Intervals
Sample mean vs Population mean
CI: > 30 samples
CI: < 30 samples
> mean(Latency) - qt(0.975,9)*sd(Latency)/sqrt(10)
[1]
> mean(Latency) + qt(0.975,9)*sd(Latency)/sqrt(10)
[1]
For the latency data, m = 10, a = 0.05:
(17.26, 21.94)
Raj Jain, “The Art of Computer Systems Performance Analysis,” Wiley, 1991.

10. Depth Resistance
Scatter and Line Plots
Resistance profile of doped silicon epitaxial layer
Expect linear resistance increase as depth increases
> data=read.table("xyscatter.dat", header=T)
> attach(data)
> model = lm(Resistance ~ Depth)
> summary(model)
Residuals:
Min Q Median Q Max
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)
Depth e-07 ***
---
Signif. codes: 0 `***' `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1
Residual standard error: on 8 degrees of freedom
Multiple R-Squared: , Adjusted R-squared:
F-statistic: on 1 and 8 DF, p-value: 1.249e-07
> plot(Depth,Resistance,main=”Epi Layer Resistance”,xlab=”Depth,
+ microns”,ylab=”Resistance, Mohms”,pch=19,
+ cex.main=1.5,cex.axis=1.5,,cex.lab=1.5)
> abline( , )
> error=Resistance-( *Depth)
> plot(Depth,error,main=”Residual Plot”,xlab=”Depth, micron”,
+ ylab=”Error, Mohms”,cex.main=1.5,cex.axis=1.5,pch=19,cex.lab=1.5)
> abline(0,0)

11. Linear Regression Statistics
model = lm(Resistance ~ Depth)
summary(model)
Residuals:
Min Q Median Q Max
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)
Depth e-07 ***
---
Signif. codes: 0 `***' `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1
Residual standard error: on 8 degrees of freedom
Multiple R-Squared: , Adjusted R-squared:
F-statistic: on 1 and 8 DF, p-value: 1.249e-07

12. Validating Residuals
Errors are marginally normally distributed due to “tails”
> qqnorm(error, pch=19,cex.lab=1.5,cex.axis=1.5,cex.main=1.5)
> qqline(error)

13. Comparing Two Sets of Data
Example: Consider two wireless different access points. Which one is faster?
Inputs: same set of 10 messages communicated through both access points.
Response (usecs):
Latency1 Latency2
Approach:
Take difference of data and determine CI of difference.
If CI straddles zero, cannot tell which access point is faster.
> data=read.table("compare.dat", header=T)
> data
Latency1 Latency2
> diff=Latency1-Latency2
> mean(diff)-qt(0.975,9)*sd(diff)/sqrt(10)
[1]
> mean(diff)+qt(0.975,9)*sd(diff)/sqrt(10)
[1]
CI95% = (-1.27, 2.87) usecs
Confidence interval straddles zero.
Thus, cannot determine which is faster with 95% confidence

14. Plots with error bars
Execution time of SuperLU linear system solution on parallel computer
Ax = b
For each p, ran problem multiple times with same matrix size but different values
Determined mean and CI for each p to obtain curve and error intervals
# Load Hmisc library
> library("Hmisc")
# Read data from file
> data <- read.table("demo.data", header=T)
# Display the data on screen
> data
x y delta
attach(data)
# Plot dashed line curve on screen with error bars
> errbar(x, y, y-delta, y+delta, xlab="Number of Processors, p", ylab="Execution Time, msecs")
> lines(x, y, type="l", lty=2)

15. How to determine O(n) > model = lm(t ~ poly(p,4))
> summary(model)
Call:
lm(formula = t ~ poly(p, 4))
Residuals:
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) e-10 ***
poly(p, 4) e-10 ***
poly(p, 4) e-08 ***
poly(p, 4) e-05 ***
poly(p, 4)
---
Signif. codes: 0 `***' `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1
Residual standard error: on 4 degrees of freedom
Multiple R-Squared: 1, Adjusted R-squared:
F-statistic: 2.38e+04 on 4 and 4 DF, p-value: 5.297e-09
> model = lm(t ~ poly(p,4))
> summary(model)
Call:
lm(formula = t ~ poly(p, 4))
Residuals:
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) e-10 ***
poly(p, 4) e-10 ***
poly(p, 4) e-08 ***
poly(p, 4) e-05 ***
poly(p, 4)
---
Signif. codes: 0 `***' `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1
Residual standard error: on 4 degrees of freedom
Multiple R-Squared: 1, Adjusted R-squared:
F-statistic: 2.38e+04 on 4 and 4 DF, p-value: 5.297e-09

16. R2 – Coefficient of Determination
SSE around the mean is
SST = ∑ (yi – mean(y))2 = ∑(yi2) – n(mean(y)2) = SSY -SS0
SSE around the model is
SSE = ∑ei2
SSR = SST – SSE
R2 = SSR/SST = (SST-SSE)/SST
R2 is a measure of how good the model is.
The closer R2 is to 1 the better.
Example: Let SST = 1499 and SSE = 97.
Then R2 = 93.5%

17. Using the t-test Consider the following data (“sleep.R”) extra group
File “sleep.R” from /usr/lib/R/library/base/data/
"sleep" <-
structure(list(extra = c(0.7, -1.6, -0.2, -1.2, -0.1, 3.4, 3.7,
0.8, 0, 2, 1.9, 0.8, 1.1, 0.1, -0.1, 4.4, 5.5, 1.6, 4.6, 3.4),
group = structure(c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2), .Label = c("1", "2"), class = "factor")),
.Names = c("extra", "group"), row.names = c("1", "2", "3", "4",
"5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15",
"16", "17", "18", "19", "20"), class = "data.frame")
To read into an R session:
> source("sleep.R")
> sleep
extra group
<more data>
From “Introduction to R”,

18. T.test result
> t.test(extra ~ group, data = sleep)
Welch Two Sample t-test
data: extra by group
t = , df = , p-value =
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
sample estimates:
mean of x mean of y
> data(sleep)
> plot(extra ~ group, data = sleep)
> ## Traditional interface
> with(sleep, t.test(extra[group == 1], extra[group == 2]))
Welch Two Sample t-test
data: extra[group == 1] and extra[group == 2]
t = , df = , p-value =
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
sample estimates:
mean of x mean of y
> ## Formula interface
> t.test(extra ~ group, data = sleep)
data: extra by group
mean in group 1 mean in group 2
p-value is smallest 1- confidence where null hyp. not true.
p-value = means difference not 0 above 92%

19. 2k Factorial Design y = q0 + qAxA + qBxB + qABxAB (k=2)
SST = total variation around the mean
= ∑ (yi – mean(y))2
= SSA+SSB+SSAB
where SSA = 22qA2
Note: var(y) = SST/(n-1)
Fraction of variation explained by A = SSA/SST

20. 2k Design Cache Factor Levels Experiment Design Address Trace Misses
Line Length (L) , 512 words
No. Sections (K) , sections
Control Method (C) multiplexed, linear
Cache
Experiment Design
Address Trace
Misses
L K C Misses
mux
mux
mux
mux
lin
lin
lin
lin
Are all factors needed?
If a factor has little effect on the variability of the output, why study it further?
Method?
a. Evaluate variation for each factor using only two levels each
b. Must consider interactions as well
Encoded Experiment Design
L K C Misses
> data=read.table("2k.data", header=T)
> data
L K C Misses
mux
mux
mux
mux
lin
lin
lin
lin
> attach(data)
> L=factor(L)
> K=factor(K)
> C=factor(C)
> analysis=aov(Misses~L*K*C)
Interaction: effect of a factor dependent on the levels of another

21. 2k Design Analyze Results (Sign Table) Obtain Reponses
I L K C LK LC KC LKC Miss.Rate
L K C Misses
> analysis
Call:
aov(formula = Misses ~ L * K * C)
Terms:
L K C L:K L:C K:C L:K:C
Sum of Squares
Deg. of Freedom
Estimated effects may be unbalanced
> summary(analysis)
Df Sum Sq Mean Sq L K C
L:K
L:C
K:C
L:K:C
> SSx=c(800,200,3200,200,32,72,8)
> SST=sum(SSx)
> Percent.Variation=100*SSx/SST
> Percent.Variation
[1]
qi:
= 1/∑(signi*Responsei)
SSL = 23q2L = 800
SST = SSL+SSK+SSC+SSLK+SSLC+SSKC+SSLKC =
= 4512
%variation(L) = SSL/SST = 800/ = 17.7%

22. Full Factorial Design Model: yij = m+ai + bj + eij
Effects computed such that ∑ai = 0 and ∑bj = 0
m = mean(y..)
aj = mean(y.j) – m
bi = mean(yi.) – m
Experimental Errors
SSE = ei2j
SS0 = abm2
SSA= b∑a2
SSB= a∑b2
SST = SS0+SSA+SSB+SSE

23. Full-Factor Design Example
Determination of the speed of light
Morley Experiments
Factors: Experiment No. (Expt)
Run No. (Run)
Levels: Expt – 5 experiments
Run – 20 repeated runs
Expt Run Speed
<more data>
> mm <- read.table("morley.tab") # Get file from
# usr/lib/R/library/base/data
> mm
Expt Run Speed
<95 more lines>
> attach(mm)
> Expt <- factor(Expt) # Experiment is a factor with levels 1, 2, ..., 5
> Run <- factor(Run) # Run is a factor with levels 1, 2, ..., 20
> Plot a boxplot of each factor
> plot(Expt, Speed, main="Speed of Light Data (units)",
xlab="Experiment No.")
> fm <- aov(Speed~Run+Expt, data=mm) # Determine ANOVA
> summary(fm) # Display ANOVA of factors
Df Sum Sq Mean Sq F value Pr(>F)
Run
Expt **
Residuals
---
Signif. codes: 0 `***' `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1

24. Box Plots of Factors

25. Two-Factor Full Factorial
> fm <- aov(Speed~Run+Expt, data=mm) # Determine ANOVA
> summary(fm) # Display ANOVA of factors
Df Sum Sq Mean Sq F value Pr(>F)
Run
Expt **
Residuals
---
Signif. codes: 0 `***' `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1
Conclusion: Data across experiments has acceptably small variation, but variation within runs is significant