Power Analysis How confident are we that we can detect an important difference, if it exists? Power = 1 - , where ES = Effect Size Used to design an experiment (a priori) –The size of an experiment is often limited by factors other than statistics –Nonetheless, it is good experimental technique to try to estimate the degree of precision that will be attained and to present this information as part of the proposal for the experiment Evaluate a completed experiment (a posteriori ) –May provide justification for publishing nonsignificant results –Guide for improving experimental technique in the future

Power Analysis Sources of input for power analysis –educated guesses derived from theory –results of previous studies reported in literature –pilot data Effect Size – several options –minimal practical significance –educated guess of the true underlying effect Questions of interest –number of replications –number of subsamples –plot size –detectable difference

Replications needed to estimate a mean For a single population, you may want to determine the sample size needed to obtain a given level of precision Recall that Rearrange the formula r is the number of replications t is the critical t with r-1 df d is the limit of the confidence interval

Example In a preliminary trial, a sample of 5 replicates has a standard deviation of 3.4 units You would like to conduct an experiment to estimate the mean within two units of the true mean with a confidence level of 95% But we have a problem here... We need to know how many reps in order to calculate degrees of freedom So it is: –pick –plug –adjust

Calculations Know your t table! In Excel, =T.INV.2T(0.05,4)  2.78 (2-tailed distribution) In Kuehl, use  =0.025 = 4  2.78 (Prob. of t  t , ) Given s = 3.4, d = 2,  = 0.05 Try starting with r = 5 r  ( * )/2 2 = 22.3 rdf tcalculated r r = 14 Additional iterations…

Detecting differences between means may be expressed as percentage of mean or on actual scale Wherer=the number of replications t 1 =t at the significance level for the test t 2 =t at 2(1-P), where P is the selected probability of obtaining a significant result (power) (note that 1-P is , the probability of a type II error)  =standard deviation (or CV%) d=the difference that it is desired to detect

What is meaningful? If it can be established that the new is superior to the old by at least some stated amount, say 20%, then we will have discovered a useful result If the experiment shows no significant difference, we will be discouraged from further investigation The experiment should be large enough to ensure a meaningful difference.

For example: We have two varieties to compare. A previous experiment with these treatments found a CV of 11% How many replications would be needed to detect a difference of 20% with a probability of.85 using a 5% significance level test? r > 2(t 1 + t 2 ) 2 CV 2 / d 2 r > 2( ) 2 (11) 2 /(20) 2 r > 7.76 ~ 8 first pick 4 reps (6 df) r > 2( ) 2 (11) 2 /(20) 2 r > 6.28 ~ 7 then test to see if 8 reps (14 df) is correct t 1 =T.INV.2T(0.05,6)=2.447 t 2 =T.INV.2T (0.30,6)=1.134 then test to see if 7 reps (12 df) is correct r > 6.44 ~ 7

Number of reps for the ANOVA The ANOVA may have more than two treatments, so we must consider differences among multiple means Power curves are commonly used –see Kuehl pg 63 for more information For this class, we will often get approximate estimates of the number of reps needed using the formula for two means –use the error term from ANOVA (MSE) to estimate s 2 –use the appropriate degrees of freedom for the MSE df = #treatments*(r-1) for a CRD df = (#treatments-1)(r-1) for an RBD

Power curve for ANOVA - example

SAS Power Calculations PROC POWER PROC GLMPOWER SAS power and sample size application –Stand alone desktop application that utilizes PROC POWER and PROC GLMPOWER through a user friendly interface

Determining Plot Size Factors that affect plot size Type of crop Type of experiment Phase of the research program Variability of the experimental site Presence and nature of border effects Type of machinery to be used Number and type of treatments Land area available Cost

Increasing Plot Size Factor Small plots Large Plots Soil variability Uniform Heterogeneous Crop Turf --Cereals -- Row crops -- Trees -- Pasture Research phase Early Late Experiment type Breeding -- Fertilizer -- Tillage -- Irrigation Machinery None Research Farm Scale Factors Affecting Plot Size

Effect on Variability Variability per plot decreases as plot size increases But large plots may yield higher experimental error because of larger more variable area for the experiment Very small plots are highly variable because: –Losses at harvest and measurement errors have a greater effect –Reduced plant numbers –Competition and border effects are greater

How Plot Size Affects Variability Variability Size

Plot Size ‘Rule of Thumb’ There are some lower limits: –Should be large enough to permit removal of borders with enough left over to harvest and measure adequately –Should be large enough to handle the machinery needed Once the plots are large enough to be handled conveniently, precision is increased faster by increasing the number of replications

Smith’s Soil Variability Index Vx =Vx = VxbVxb Where:V= variance of a unit plot V x = variance, on a per unit basis, of plots formed from x adjacent units x=plot size in multiples of adjacent unit plots b=index of soil variability

Smith’s Soil Variability Index The index can vary from 0 to 1 When b=0 then V x = V This means there is no relation between variance and plot size. Adjacent plots are completely correlated. Nothing can be gained from larger plots. This is the case when the soil is highly uniform. When b=1 then V x = VxVx This means that the units that make up the plots are independent of each other. Increasing plot size will reduce variance. This is the case when the soil is highly heterogeneous. Vx =Vx = VxbVxb

Effect of plant sample size However, can only obtain b=0 when plant sample size is above a minimal level For large plants, sample size may be more critical than soil variability in determining optimal plot size

Variability Index Calculation log V x = log V - b log x y = a + bx Log of plot size Log of variance per plot a (intercept) b (slope) yxyx Vx =Vx = VxbVxb

Raw data from Uniformity Trial Combine yields of plots from adjacent rows (2x1)

1x x x x x x x x x x x x x x x x x x x CombinationxLog(x)V x Log(V x )

Smith’s Index of Variability log V x = log x log V x = log V - b log x y = a + bx

Is there an easier way? Examination of a large number of data sets indicated that a value of b=0.5 may serve as a reasonable approximation. “Finagles” constant b=0.5

Optimum Plot Size Optimum Size will either minimize cost for a fixed variance or minimize variance for a fixed cost You must know certain costs: T = K 1 + K 2 x Where:T =total cost per plot ($ or time) K 1 =cost per plot ($ or time) that is independent of plot size K 2 =cost per plot ($ or time) that depends on plot size x =number of unit plots

Optimum Plot Size n Knowing those costs, Smith found the optimum plot size to be: x opt = bK 1 (1-b)K 2 Where:T =total cost per plot ($ or time) K 1 =cost per plot ($ or time) that is independent of plot size K 2 =cost per plot ($ or time) that depends on plot size x =number of unit plots b =Smith’s index of soil variability

Optimum Plot Size So in our example, we found b to be 0.70 and our unit plot area was 5m x 2m or 10m 2 Assume K 1 = $3.00 and K 2 = $5.00 for 10 m 2 = (0.70)(3.00) = 1.4 (1-0.70)(5.00) Area = (1.4)(10) = 14.0 m 2 2 m 7 m x opt = bK 1 (1-b)K 2

Convenient Plot Size Smith’s optimum plot size was based on soil variability and cost. Hathaway developed a formula based on soil variability, the size of difference to be detected, and significance level of the test. x b = 2(t 1 + t 2 ) 2  2 rd 2

Convenient Plot Size Where:x=number of units of plots of size x (plot size) b=Smith’s coefficient of soil variability t 1 =t at the significance level for the test t 2 =t in the t table at 2(1-P), where P is the selected probability of obtaining a significant result  2 =variance from a previous experiment (may also use the CV 2 ) r=number of replications d=difference to be detected (may be an absolute amount or expressed as percentage of the mean) x b = 2(t 1 + t 2 ) 2  2 rd 2

For Example... A variety trial of 50 selections in a randomized block design with three blocks What size plot do we need to detect a difference of 25% of the mean with an 80% probability of obtaining a significant result using a 5% significance level test? Previous experiment had a CV of 11%

x b = 2(t 1 + t 2 ) 2 CV 2 rd 2 error df = (3-1)(50-1) = 98 t 1 = t 0.05(98)) = t 2 = t 0.40(98) = 0.845{t 2(1-P) = t 2(1-0.80) = t 0.40 } b = 0.70 CV = 11% Then: x b = 2( ) 2 (11) 2 = (3)(25) x b = or x.70 = x = /.70 = basic units

Convenient Plot Size Since the basic plot has an area of 10 m 2, the required plot size would be: (10.00)(1.0475) = m 2 2 m 5.24 m

The Detectable Difference Using Hathaway’s formula for convenient plot size and solving for d 2, it is possible to compute the detectable difference possible when you know the number of reps and the plot size. d 2 = 2(t 1 + t 2 ) 2  2 rx b

An Alternative Example We want to find the difference that can be detected 80% of the time at a 5% significance level using a plot 2 m wide and 7 m long in 4 replications As before, the CV=11%, and b=0.70 d 2 = 2(t 1 + t 2 ) 2 CV 2 rx b

d 2 = 2(t 1 + t 2 ) 2 CV 2 rx b d 2 = 2( ) 2 (11) 2 (4)( ) because 2m*7m/10m 2 =1.4 = (2)(7.954)(121) = (4)(1.2656) = d = Therefore, with a plot size of 2m x 7m using 4 replications, a difference of 19.50% of the mean could be detected

Using these tools Detectable difference (% of Mean) Plot Area (m 2 ) Replication