1. VLSI Systems Design—Experiments Necessary steps: Explore the problem space Design experiment(s) Carry out experiment(s) Analyze results software packages: R, Matlab, … Report results

2. Example: design a “better” transistor What do we mean by “better”? What FACTORS influence design? --fabrication --design --environmental For which of these is there random variation?

3. Which “factors” do we want to investigate?

4. SUMMARY—15 IMPORTANT POINTS FOR EXPERIMENTERS: 1. Even careful experimentation and observation may miss important facts; new experiments may cause old conclusions to be discarded; EXPERIMENTS ARE NOT PROOFS. 2. It is just as important to report NEGATIVE results as to report POSITIVE results. The experimenter must always accurately record and thoroughly report ALL results. 3. IGNORING IMPORTANT FACTORS CAN LEAD TO ERRONEOUS CONCLUSIONS, SOMETIMES WITH TRAGIC RESULTS. 4. YOUR RESULTS ARE ONLY VALID FOR THE PART OF THE DATA-TREATMENT SPACE YOU HAVE EXPLORED; YOU CANNOT CLAIM KNOWLEDGE OF WHAT YOU HAVE NOT EXPLORED 5. An experiment is worthless unless it can be REPEATED by other researchers using the same experimental setup; experimenters have a duty to the research community to report enough about their experiment and data so that other researchers can verify their claims 6. YOU ONLY GET ANSWERS TO THE QUESTIONS YOU ASK 7. if your are going to use a (pseudo-)RANDOM NUMBER GENERATOR, make sure the output behaves enough like a sequence of TRUE RANDOM NUMBERS 8. An experiment must be repeated a SUFFICIENT NUMBER OF TIMES for the results to be attributed to more than random error 9. Choosing the CORRECT MEASURE for the question you are asking is an important part of the experimental design 10. Reporting CORRECT results, PROPERLY DISPLAYED, is an integral part of a well-done experiment 11. MISUSE OF GRAPH LABELING can lead to MISLEADING RESULTS AND INCORRECT CONCLUSIONS 12. INTERPOLATING your results to regions you have not explored can lead to INCORRECT CONCLUSIONS 13. IGNORING the “NULL HYPOTHESIS” when reporting your results can be very misleading 14. Don’t mistake CORRELATION for DEPENDENCE 15. Justify your choice of CURVE using VALID STATISTICS, not “appearance”

5. Topics Analyzing and Displaying Data –Simple Statistical Analysis –Comparing Results –Curve Fitting Designing Experiments: Factorial Designs –2 K Designs Including Replications –Full Factorial Designs Ensuring Data Meets Analysis Criteria Presenting Your Results; Drawing Conclusions

6. Example: A System System (“Black Box”) System Inputs System Outputs Factors (Experimental Conditions) Responses (Experimental Results)

7. Experimental Research Define System Define System Identify Factors and Levels Identify Factors and Levels Identify Response(s) Identify Response(s) ● Define system outputs first ● Then define system inputs ● Finally, define behavior (i.e., transfer function) ● Identify system parameters that vary (many) ● Reduce parameters to important factors (few) ● Identify values (i.e., levels) for each factor ● Identify time, space, etc. effects of interest Design Experiments Design Experiments ● Identify factor-level experiments

8. Create and Execute System; Analyze Data Define Workload Define Workload Create System Create System Execute System Execute System ● Workload can be a factor (but often isn't) ● Workloads are inputs that are applied to system ● Create system so it can be executed ● Real prototype ● Simulation model ● Empirical equations ● Execute system for each factor-level binding ● Collect and archive response data Analyze & Display Data Analyze & Display Data ● Analyze data according to experiment design ● Evaluate raw and analyzed data for errors ● Display raw and analyzed data to draw conclusions

9. Some Examples Analog Simulation –Which of three solvers is best? –What is the system? –Responses Fastest simulation time Most accurate result Most robust to types of circuits being simulated –Factors Solver Type of circuit model Matrix data structure Epitaxial growth –New method using non- linear temp profile –What is the system? –Responses Total time Quality of layer Total energy required Maximum layer thickness –Factors Temperature profile Oxygen density Initial temperature Ambient temperature

10. Basic Descriptive Statistics for a Random Sample X Mean Median Mode Variance / standard deviation Z scores: Z = (X – mean)/ (standard deviation) Quartiles, box plots Q-Q plot Note: these can be deceptive. For example, if P (X = 0) = P(X = 100) = 0.5 and P (Y = 50 ) = 1, Then X and Y have the same mean (and nastier examples can be constructed) home.oise.utoronto.ca/~thollenstein/Exploratory%20Data%20Analysis.ppt

11. SIMPLE MODELS OF DATA Ms. #Latency 122 223 319 418 515 620 726 817 919 1017 Data file “latency.dat” Example: Evaluation of a new wireless network protocol System: wireless network with new protocol Workload: 10 messages applied at single source Each message identical configuration Experiment output: Roundtrip latency per message (ms) Mean: 19.6 ms Variance: 10.71 ms 2 Std Dev: 3.27 ms Hypothesis: Distribution is N(  2 )

12. Verify Model Preconditions Check randomness Use plot of residuals around mean Residuals “appear” random Check normal distribution Use quantile-quantile (Q-Q) plot Pattern adheres consistently along ideal quantile-quantile line http://itl.nist.gov/div898/software/dataplot/refman1/ch2/quantile.pdf

13. Confidence Intervals Sample mean vs Population mean If many samples are collected, about 1 -  will contain the “true mean” CI: > 30 samples CI: < 30 samples For the latency data,  = 10, a = 0.05: (17.26, 21.94) Raj Jain, “The Art of Computer Systems Performance Analysis,” Wiley, 1991.

14. Scatter and Line Plots DepthResistance 11.689015 24.486722 37.915209 46.362388 511.830739 612.329104 714.011396 817.600094 919.022146 1021.513802 Resistance profile of doped silicon epitaxial layer Expect linear resistance increase as depth increases

15. Linear Regression Statistics (hypothesis: resistance =  0 +  1 *depth + error) model = lm(Resistance ~ Depth) summary(model) Residuals: Min 1Q Median 3Q Max -2.11330 -0.40679 0.05759 0.51211 1.57310 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -0.05863 0.76366 -0.077 0.94 Depth 2.13358 0.12308 17.336 1.25e-07 *** --- Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 Residual standard error: 1.118 on 8 degrees of freedom “variance of error: (1.118) 2 ” Multiple R-Squared: 0.9741, Adjusted R-squared: 0.9708 F-statistic: 300.5 on 1 and 8 DF, p-value: 1.249e-07 “evidence this estimate valid” “reject hypotheses  0 = 0,  1 = 0” (Using R system; based on http://www.stat.umn.edu/geyer/5102/examp/reg.html

16. Validating Residuals Errors are marginally normally distributed due to “tails”

17. Comparing Two Sets of Data Example: Consider two different wireless access points. Which one is faster? Inputs: same set of 10 messages communicated through both access points. Response (usecs): Latency1 Latency2 22 19 23 20 19 24 18 20 15 14 20 18 26 21 17 17 19 17 17 18 Approach: Take difference of data and determine CI of difference. If CI straddles zero, cannot tell which access point is faster. CI 95 % = (-1.27, 2.87) usecs Confidence interval straddles zero. Thus, cannot determine which is faster with 95% confidence

18. Plots with error bars Execution time of SuperLU linear system solution on parallel computer Ax = b For each p, ran problem multiple times with same matrix size but different values Determined mean and CI for each p to obtain curve and error intervals Matrix density p

19. Curve Fitting > model = lm(t ~ poly(p,4)) > summary(model) Call: lm(formula = t ~ poly(p, 4)) Residuals: 1 2 3 4 5 6 7 8 9 -0.4072 0.7790 0.5840 -1.3090 -0.9755 0.8501 2.6749 -3.1528 0.9564 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 236.9444 0.7908 299.636 7.44e-10 *** poly(p, 4)1 679.5924 2.3723 286.467 8.91e-10 *** poly(p, 4)2 268.3677 2.3723 113.124 3.66e-08 *** poly(p, 4)3 42.8772 2.3723 18.074 5.51e-05 *** poly(p, 4)4 2.4249 2.3723 1.022 0.364 --- Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 Residual standard error: 2.372 on 4 degrees of freedom Multiple R-Squared: 1, Adjusted R-squared: 0.9999 F-statistic: 2.38e+04 on 4 and 4 DF, p-value: 5.297e-09

20. Model Validation: y’ = ax + b R 2 – Coefficient of Determination “How well does the data fit your model?” What proportion of the “variability” is accounted for by the statistical model? (what is ratio of explained variation to total variation?) Suppose we have measurements y 1, y 2, …, y n with mean m And predicted values y 1 ’, y 2 ’, …, y n ’ (y i ’ = ax i + b = y i + e i ) SSE = sum of squared errors = ∑ (y i – y i ’) 2 = ∑e i 2 SST = total sum of squares =∑ (y i – m) 2 SSR = SST – SSE = residual sum of squares = ∑ (m – y i ’) 2 R 2 = SSR/SST = (SST-SSE)/SST R 2 is a measure of how good the model is. The closer R 2 is to 1 the better. Example: Let SST = 1499 and SSE = 97. Then R 2 = 93.5% http://www-stat.stanford.edu/~jtaylo/courses/stats191/notes/simple_diagnostics.pdf

21. Using the t-test extra group 1 0.7 1 2 -1.6 1 3 -0.2 1 4 -1.2 1 5 -0.1 1 6 3.4 1 7 3.7 1 8 0.8 1 9 0.0 1 10 2.0 1 11 1.9 2 12 0.8 2 13 1.1 2 14 0.1 2 15 -0.1 2 16 4.4 2 17 5.5 2 18 1.6 2 19 4.6 2 20 3.4 2 Consider the following data (“sleep.R”) From “Introduction to R”, http://www.R-project.org

22. T.test result > t.test(extra ~ group, data = sleep) Welch Two Sample t-test data: extra by group t = -1.8608, df = 17.776, p-value = 0.0794 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -3.3654832 0.2054832 sample estimates: mean of x mean of y 0.75 2.33 p-value is smallest 1- confidence where null hypothesis. not true. p-value = 0.0794 means difference not 0 above 92%

23. Factorial Design What “factors” need to be taken into account? How do we design an efficient experiment to test all these factors? How much do the factors and the interactions among the factors contribute to the variation in results? Example: 3 factors a,b,c, each with 2 values: 8 combinations But what if we want random order of experiments? What if each of a,b,c has 3 values? Do we need to run all experiments? http://www.itl.nist.gov/div898/handbook/pri/section3/pri3332.htm

24. Standard Procedure-Full Factorial Design (Example) Variables A,B,C: each with 3 values, Low, Medium, High (coded as -1,0,1) “Signs Table”: ABC 1 2+1 3 +1 4+1 5 +1 6 +1 7+1 8 1.Run the experiments in the table (“2 level, full factorial design”) 2.Repeat the experiments in this order n times by using rows 1,…,8,1,…,8, … (“replication”) 3.Use step 2, but choose the rows randomly (“randomization”) 4.Use step 4, but add some “center point runs”, for example, run the case 0,0,0, then use 8 rows, then run 0,0,0, …finish with a 0,0,0 case In general, for 5 or more factors, use a “fractional factorial design”

25. 2 k Factorial Design Example: k = 2, factors are A,B, and X’s are computed from the signs table: y = q 0 + q A x A + q B x B + q AB x AB SST = total variation around the mean = ∑ (y i – mean) 2 = SSA+SSB+SSAB where SSA = 2 2 q A 2 (variation allocated to A), and SSB, SSAB are defined similarly Note: var(y) = SST/( 2 k – 1) Fraction of variation explained by A = SSA/SST

26. Example: 2 k Design Are all factors needed? If a factor has little effect on the variability of the output, why study it further? Method? a. Evaluate variation for each factor using only two levels each b. Must consider interactions as well Interaction: effect of a factor dependent on the levels of another L K C Misses 32 4 mux 512 4 mux 32 16 mux 512 16 mux 32 4 lin 512 4 lin 32 16 lin 512 16 lin Factor Levels Line Length (L) 32, 512 words No. Sections (K) 4, 16 sections Control Method (C) multiplexed, linear Experiment Design Cache Address Trace Misses L K C Misses -1 -1 -1 1 -1 -1 -1 1 -1 1 1 -1 -1 -1 1 1 -1 1 -1 1 1 1 1 1 Encoded Experiment Design www.stat.nuk.edu.tw/Ray-Bing /ex-design/ex-design/ExChapter6.ppt

27. I L K C LK LC KC LKC Miss.Rate (y j ) 1 -1 -1 -1 1 1 1 -1 14 1 1 -1 -1 -1 -1 1 1 22 1 -1 1 -1 -1 1 -1 1 10 1 1 1 -1 1 -1 -1 -1 34 1 -1 -1 1 1 -1 -1 1 46 1 1 -1 1 -1 1 -1 -1 58 1 -1 1 1 -1 -1 1 -1 50 1 1 1 1 1 1 1 1 86 Analyze Results (Sign Table) q i : 40 10 5 20 5 2 3 1 = 1/ ∑ (sign i *Response i ) SSL = 2 3 q 2 L = 800 SST = SSL+SSK+SSC+SSLK+SSLC+SSKC+SSLKC = 800+200+3200+200+32+72+8 = 4512 %variation(L) = SSL/SST = 800/4512 = 17.7% L K C Misses -1 -1 -1 14 1 -1 -1 22 -1 1 -1 10 1 1 -1 34 -1 -1 1 46 1 -1 1 58 -1 1 1 50 1 1 1 86 Obtain Reponses Example: 2 k Design (continued) Ex: y 1 = 14 = q 0 – q L –q K –q C + q LK + q LC + q KC – q LKC Solve for q’s http://www.cs.wustl.edu/~jain/cse567-06/ftp/k_172kd/sld001.htm

28. Full Factorial Design Model: y ij = m+a i + b j + e ij Effects computed such that ∑a i = 0 and ∑b j = 0 m = mean(y..) a j = mean(y. j ) – m bi = mean(y i.) – m Experimental Errors SSE =  e i 2 j SS0 = abm 2 SSA= b∑a 2 SSB= a∑b 2 SST = SS0+SSA+SSB+SSE

29. Full-Factorial Design Example Determination of the speed of light Morley Experiments Factors: Experiment No. (Expt) Run No. (Run) Levels: Expt – 5 experiments Run – 20 repeated runs Expt Run Speed 001 1 1 850 002 1 2 740 003 1 3 900 004 1 4 1070 019 1 19 960 020 1 20 960 021 2 1 960 022 2 2 940 023 2 3 960 096 5 16 940 097 5 17 950 098 5 18 800 099 5 19 810 100 5 20 870

30. Box Plots of Factors

31. Two-Factor Full Factorial > fm <- aov(Speed~Run+Expt, data=mm) # Determine ANOVA > summary(fm) # Display ANOVA of factors Df Sum Sq Mean Sq F value Pr(>F) Run 19 113344 5965 1.1053 0.363209 Expt 4 94514 23629 4.3781 0.003071 ** Residuals 76 410166 5397 --- Signif. codes: 0 `***' 0.001 `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 Conclusion: Data across experiments has acceptably small variation, but variation within runs is significant

32. Visualizing Results: Tufte’s Principles Have a properly chosen format and design Use words, numbers, and drawing together Reflect a balance, a proportion, a sense of relevant scale Display an accessible complexity of detail Have a story to tell about the data Draw in a professional manner Avoid content-free decoration, including “chart junk”

33. Back to the transistor: What factors are there? Which ones do we want to investigate? How should we define our experiments? What role will randomness play? (simulation/actual) How should we report the results?