Interval Estimation of mean response 𝑬 𝒀 𝒉 , 𝒐𝒓 𝒀 𝒉 and single response 𝒀 𝒉 (new) when 𝑋= 𝑋 ℎ A common objective in regression analysis is to estimate the mean of response variable Y for a given independent variable X. Let Xh denote the level of X that we are interested. The mean response is denoted by E(Yh) or Yh hat, and the single response is denoted by Yh-hat (new)

Mean response vs. single response 𝑌 1 for a given 𝑋 1 = mean + random error 𝑌 2 for a given 𝑋 2 = mean + random error The mean interval single Y ±𝒎𝒂𝒓𝒈𝒊𝒏 𝒆𝒓𝒓𝒐𝒓 is larger than The mean interval 𝝁±𝒎𝒂𝒓𝒈𝒊𝒏 𝒆𝒓𝒓𝒐𝒓 𝑋 1 𝑋 2 Here is a picture displaying both mean response and the single response variables. Recall that in topic1 when talk about two ingredients in regression model, we referred to regression model as a “subsampling model” because the data set of response variable Y consists of many subsamples (or slices), one subsample per each given value of X. The mean of Y are linearly related to X, while single Y randomly varies around its mean. The mean value of Yh, or E(Yh) given X=Xh, is expected to be on the line, and can be estimated with a linear function b0+b1Xh. The actual value of a single response variable Y is the value scattering around the mean, and can be represented as mean + random error. The variance of the single response variable is then variance of the mean + the variance of the random error. [B] As a result, the confidence interval of the single response should be larger than of the mean response. Predict the mean response of Y on X Predict the single response of Y on X 𝑬 𝒀 𝒉 𝑜𝑟 𝜇 𝒉 Predict in the same manner, Same value; But different precision. 𝒀 𝒉

The best estimates of β1 and β0 given the data (X, Y ) are: Recall again from topic 1 that, the formula of parameter beta1 estimator b1 is a linear combination of Y. So b1 follows normal distribution. Similarly, b0 is a linear combination of Y and also follows normal distribution. As a result, the estimate of Y when X=Xh, denoted by Yh-hat, is bo+b1Xh is a linear combination of Y. Question1: Does Yh_hat follow normal distribution? Quesiton2: are b0 and b1 independent two independent variables? Recall that The best estimates of β1 and β0 given the data (X, Y ) are: 𝑏 1 = Σ(𝑋− 𝑋 )(𝑌− 𝑌 ) Σ 𝑋− 𝑋 2 = SS XY SS X 𝐸 𝑏 1 = 𝛽 1 𝑎𝑛𝑑 𝑉𝑎𝑟 𝑏 1 = 𝜎 2 1 Σ 𝑋 𝑖 − 𝑋 2 = 𝛴 𝑐 𝑖 𝑌 𝑖 𝑏 0 = 𝑌 − 𝑏 1 𝑋 = Σ Y i n −𝛴 𝑐 𝑖 𝑋 𝑌 𝑖 =Σ 𝑑 𝑖 𝑌 𝑖 𝐸 𝑏 0 = 𝛽 0 𝑎𝑛𝑑 𝑉𝑎𝑟 𝑏 0 = 𝜎 2 [ 1 𝑛 + 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 ] Hence, 𝑌 ℎ = 𝑏 0 + 𝑏 1 𝑋 ℎ is a linear combination of the observations 𝑌 𝑖 Question1: Does 𝑌 ℎ follow normal distribution? Question2: are 𝑏 0 𝑎𝑛𝑑 𝑏 1 independent?

The best estimates of β1 and β0 given the data (X, Y ) are: Since Yh-hat is linear combination of some independent and Normal variables Yi. Yh_hat follows normal distribution. For question2, because b0 and b1 are each based on the linear combination of the same Yi, they are not independent. In fact, one can derive bo from b1, or vise versa. Recall that The best estimates of β1 and β0 given the data (X, Y ) are: 𝑏 1 = Σ(𝑋− 𝑋 )(𝑌− 𝑌 ) Σ 𝑋− 𝑋 2 = SS XY SS X 𝐸 𝑏 1 = 𝛽 1 𝑎𝑛𝑑 𝑉𝑎𝑟 𝑏 1 = 𝜎 2 1 Σ 𝑋 𝑖 − 𝑋 2 = 𝛴 𝑐 𝑖 𝑌 𝑖 𝑏 0 = 𝑌 − 𝑏 1 𝑋 = Σ Y i n −𝛴 𝑐 𝑖 𝑋 𝑌 𝑖 =Σ 𝑑 𝑖 𝑌 𝑖 𝐸 𝑏 0 = 𝛽 0 𝑎𝑛𝑑 𝑉𝑎𝑟 𝑏 0 = 𝜎 2 [ 1 𝑛 + 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 ] Hence, 𝑌 ℎ = 𝑏 0 + 𝑏 1 𝑋 ℎ is a linear combination of the observations 𝑌 𝑖 Question1: Does 𝑌 ℎ follow normal distribution? Yes Question2: are 𝑏 0 𝑎𝑛𝑑 𝑏 1 independent? No

Prediction of the mean response The mean estimate, defined by E(Yh) or Yh_hat, can be obtained from the linear function, b0+b1Xh. This estimate is an unbiased estimator of E{Yh}. The variability of the sampling distribution of Yh-hat is affected by how far Xh is from X-bar, through the term (Xh-Xbar)^2. The further from Xbar is Xh, the greater is the quality (Xh-Xbar)^2 and the larger is the variance of this estimation Yh hat. An intuitive explanation of this effect can be found in the picture with two sample regression lines, based on two samples for the same set of X values. The two regression lines are assumed to go though the same (Xhar, Ybar) point. Note that at X1, near Xbar, the fitted values yhat1 for the two sample regression lines are close to each other. At a far away point X2, the fitted values Yhat2 differ substantially. Thus, the variation in the Yhat valules from sample to sample will be greater when Xh is far from the mean than near the mean. Prediction of the mean response 𝑌 ℎ = 𝑏 0 + 𝑏 1 𝑋 ℎ For normal error (𝜀) regression model, 𝑌 ℎ ~ 𝑁𝑜𝑟𝑚𝑎𝑙, with mean and variance: 𝐸 𝑌 ℎ =𝐸 𝑌 ℎ = 𝜇 ℎ 𝜎 2 𝑌 ℎ = 𝜎 2 [ 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 ] = 𝛽 0 + 𝛽 1 𝑋 ℎ 𝑌 ℎ is normal because b0 + b1Xh is a linear combination of independent, normal Yi’s. Its variance is affected by how far 𝑋 ℎ is from 𝑋 , through the term 𝑋 ℎ − 𝑋 2 . Estimation is more precise near X¯ . 𝑌 2 𝑌 1 𝑋 1 𝑋 𝑋 2

Prediction of the mean response When replacing sigma^2 with MSE, the t statistic follows t(n-2 ) distribution. All inferences concerning the mean response estimate, E(Yh) are carried out the usual fashion with the t distribution. Thus, we are using the T method to study the mean response estimate. Prediction of the mean response 𝑌 ℎ = 𝑏 0 + 𝑏 1 𝑋 ℎ For normal error (𝜀) regression model, 𝑌 ℎ ~ 𝑁𝑜𝑟𝑚𝑎𝑙, with mean and variance: 𝐸 𝑌 ℎ =𝐸 𝑌 ℎ = 𝜇 ℎ 𝜎 2 𝑌 ℎ = 𝜎 2 [ 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 ] When replace 𝜎 2 with MSE, 𝑠 2 𝑌 ℎ =𝑀𝑆𝐸 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 = s 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 Therefore, it follows that 𝑌 ℎ −𝐸 𝑌 ℎ 𝑠 𝑌 ~ 𝑡 𝑛−2

Prediction confidence interval of mean response, 𝑬 𝒀 𝒉 The confidence interval in the form of estimate plus or minus margin error, or t times standard error of the estimate Where the standard error can be computed with the equation given. s of yh-hat is the standard error of the mean response at Xh, and s is the standard error of the residuals. Prediction confidence interval of mean response, 𝑬 𝒀 𝒉 𝑌 ℎ −𝐸 𝑌 ℎ 𝑠 𝑌 ℎ ~ 𝑡 𝑛−2 The confidence interval of 𝐸 𝑌 ℎ 𝑌 ℎ ±𝑡 1− 𝛼 2 ;𝑛−2 𝑠{ 𝑌 ℎ } 𝑤ℎ𝑒𝑟𝑒 𝑠 𝑌 ℎ 2 =𝑠 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 𝑠{ 𝑌 ℎ } is the “standard error of the mean response value at 𝑋= 𝑋 ℎ ” 𝑠 is the “standard error of the residuals”

Prediction of single response 𝒀 𝒉 𝒏𝒆𝒘 The confidence interval of the estimated mean response E(Yh), given Xh, gives a upper and lower bound of the mean response. The single response Yh is based on the mean, and predicted as mean+a random error. The prediction of single response value is denoted as Yh hat (new). As shown in the picture, from the mean corresponding to the upper and lower limit of the confidence interval, there comes two probability distribution of Y, corresponding to lowest and highest possible value of mean. Any distribution between the left and right normal distribution is possible. The variance of the means is denoted by sigma^2 of the Yh hat. Suppose we now locate a distribution by fixing the mean, a single value still varies within that whole distribution. It could be here, here, here, etc, this part of variation is denoted by sigma^2. We say that the variance of predicting a single value comes from the variance in predicting the mean or predicting one distritbution, and then the variance of predicting a single value which that distribution. Prediction of single response 𝒀 𝒉 𝒏𝒆𝒘 𝑌 ℎ −𝑡 𝑠{ 𝑌 ℎ } 𝑌 ℎ +𝑡 𝑠{ 𝑌 ℎ } 𝜎 2 Y ℎ 𝑛𝑒𝑤 = 𝜎 2 𝑌 ℎ + 𝜎 2 The variance of prediction = variance in possible location of the distribution + variance within the distribution

We estimate the prediction variance σ2 The variance of predicting the single value can be estimated using the sample standard error, denoted by s{pred} The test statistic has a t distribution. We estimate the prediction variance σ2 as: {Yh(new)} 𝑠 𝑝𝑟𝑒𝑑 2 = 𝑠 𝑌 ℎ 2 + 𝑠 2 =𝑠 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 +1 For normal error regression model Yh(new) − Yˆh ∼ t (n-2) s{pred} 𝑠{𝑝𝑟𝑒𝑑} is the “standard error for predicting one new response value at 𝑋 ℎ .”

𝑠 𝑝𝑟𝑒𝑑 2 = 𝑠 𝑌 ℎ 2 + 𝑠 2 =𝑠 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 +1 The prediction interval of single response is sensitive if the error terms is not normal, meaning we cannot use the t method when the random error or the response variable Y is not normally distributed. Predictions are more precise near mean Xbar because the variance decreases with the deviation, Xh and Xbar. Prediction made near X=Xbar is more reliable because Xh-Xbar is smaller and the standard error is smaller. That is why that confidence interval are not always the same width for all Xh levels. It should be the narrowest when Xh is near the Xbar, and wider further apart. Prediction interval of single response 𝒀 𝒉 𝒏𝒆𝒘 𝑌 ℎ ±𝑡 1− 𝛼 2 ;𝑛−2 𝑠{𝑝𝑟𝑒𝑑} 𝑠 𝑝𝑟𝑒𝑑 2 = 𝑠 𝑌 ℎ 2 + 𝑠 2 =𝑠 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 +1 More sensitive to departure of normal in error terms distribution. Predictions are more precise near X¯ because σ2 decreases with |Xh − X¯ |. {Yˆh}

𝑠{𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛} is the “standard error for predicting the mean of m Occasionally, we would like to predict the mean of m new observations on Y for a given Xh value. The notation, Yh bar new, is easily confused with the previous two, Yh hat for the mean response prediction and Yh hat for the single response response. s{predmean} is the standard error for predicting the mean of m new response values. Predicting the mean of m new response values is easier (less variation) than predicting a single value, but harder (more variation) than predicting the mean response. Similar with the single response, the standard error of the mean of m responses consist of the variance between the distribution (color coded in purple), and the variance within a distribution (color coded by green). Prediction interval of mean of 𝒎 new response 𝒀 𝒉{𝒏𝒆𝒘 } not 𝑌 ℎ , 𝑜𝑟 𝑌 ℎ {𝑛𝑒𝑤} 𝑌 ℎ ±𝑡 1− 𝛼 2 ;𝑛−2 𝑠{𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛} Where: 𝑠 2 𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛 = 𝑀𝑆𝐸 𝑚 + 𝑠 2 𝑌 ℎ =𝑀𝑆𝐸[ 1 𝑚 + 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 ] Predict the mean of 𝑚 new observations on Y for a given level of the predictor variables. The variance 𝑠 2 𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛 has two components: variance between the distribution and variance within a distribution. 𝑠{𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛} is the “standard error for predicting the mean of m new response value.”

The Diamond example, if 𝑋 ℎ =0.43 , compute 1. The confidence interval for the mean predicted value 𝐸 𝑌 ℎ Where 𝑠 𝑌 ℎ 2 =𝑠 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 Yˆh ± 𝑡 𝑐 𝑠 𝑌 ℎ 2. The confidence interval for the single predicted value 𝑌 ℎ Where 𝑠 𝑝𝑟𝑒𝑑 2 = 𝑠 𝑌 ℎ 2 + 𝑠 2 =𝑠 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 +1 Yˆh ± 𝑡 𝑐 𝑠 𝑝𝑟𝑒𝑑 Now let’s see how to compute three prediction confidence interval by hand and by R. In the diamond ring example, at X=0.43, find the confidence interval for the mean predicted value, single predicted value and the mean predicted value of three diamond rings with the same weight. 3. The confidence interval for the mean price 𝑌 ℎ 𝑛𝑒𝑤 of three diamonds with the same weight (0.43) Where: 𝑠 2 𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛 = 𝑀𝑆𝐸 𝑚 + 𝑠 2 𝑌 ℎ =𝑀𝑆𝐸[ 1 𝑚 + 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 ] 𝑌 ℎ ± 𝑡 𝑐 𝑠{𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛}

Recall that in the Diamond example The lm output Recall that in the diamond example, the lm output gives a residual standard error to be 31.84, and degree of freedom of 46. Compute MSE to be the squared of the residual standard error, or 1013.8. The sample mean of the weight is computed as 0.204, sample standard deviation is 0.0568, and the sample size is 48. 𝑴𝑺𝑬= 𝒔 𝟐 = 𝟑𝟏.𝟖𝟒 𝟐 =𝟏𝟎𝟏𝟑.𝟖 𝑋 =0.204, 𝑠 𝑋 =0.0568, n=48

Recall that in the diamond example Where 𝛼=0.05, 𝑛=48, 𝑑𝑓=46 𝑟𝑜𝑢𝑛𝑑 𝑑𝑜𝑤𝑛 𝑡𝑜 40 Also recall that in the diamond example, in order to estimate the parameter beta1, we have found the t value both using t table and R. When using the t table, we round down the df to 40 from 46, and get the value of 2.021, at 95% confidence level. In general, the notation of the t value is t(1-alpha over 2 and n-2), in the case of 95% level, alpha is 5%, and the t value is denoted by t(0.975, 46). And the estimation is 2.021 using the t able or 2.013 using R. The qt function in R gives the exact value, where qt mean to the quantile value in a t distribution. Both values are Okay to use. 𝑡 1− 𝛼 2 , 𝑛−2 =𝑡 0.975, 46 Or use R =2.021 (estimation using the t table) =2.013 (exact value using R)

The Diamond example, if 𝑋 ℎ =0.43 , compute 1. The confidence interval for the mean predicted value 𝐸 𝑌 ℎ Where 𝑠 𝑌 ℎ 2 =𝑠 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 Yˆh ± 𝑡 𝑐 𝑠 𝑌 ℎ 𝑆 𝑆 𝑋 = 𝑠 𝑋 2 𝑛−1 = 0.0568 2 48−1 =0.152 𝑡 1− 𝛼 2 , 𝑛−2 =𝑡 0.975, 46 = estimation from T−table or 2.013 (exact value from R) 2.021 Now we are ready to make prediction on the average price of a diamond ring of 0.43 carat. The confidence interval for the mean prediction is Yh hat plus or minus t times standard error. The sum of squares of x, or SSx is 0.152, the t value is 2.021 using the t table, or 2.013 using R. The variance of mean prediction is computed with the formula as 362.14, then take the square root to get the standard error to be 19.03. Last, we have the confidence interval, and conclude that we are 95% confidence to predict the average price of a 0.43 carat diamond ring is at least 1302.1 dollar and at most 1378.73 dollar. In R, the ci.reg function compute all three kinds of predicting confidence intervals. The first parameter is the regression model, the second parameter, “new” is a data frame specify the Xh value , or 0.43 in example, type of m specify this is a prediction for the mean response value E(Yh hat), and alpha is the significant level or 0.05. To compute the different confidence intervals, we can change the “type” in the ci.reg function, shown next. 𝑠 𝑌 ℎ 2 =𝑠 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 = 31.84 2 1 48 + 0.43−0.204 2 0.152 =362.14 𝑠 𝑌 ℎ = 362.14 =19.03 Yˆh ± 𝑡 𝑐 𝑠 𝑌 ℎ =1340.415±2.013 19.03 =1302.1, 1378.73.

The Diamond example, if 𝑋 ℎ =0.43 , compute 2. The confidence interval for the single predicted value 𝑌 ℎ Where 𝑠 𝑝𝑟𝑒𝑑 2 = 𝑠 𝑌 ℎ 2 + 𝑠 2 =𝑠 2 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 +1 Yˆh ± 𝑡 𝑐 𝑠 𝑝𝑟𝑒𝑑 Next let’s predict the price of a single diamond ring of 0.43 carat. The confidence interval is similar, except we have a larger standard error for single value prediction. The variance of single value prediction, s^2 pred = 1375.92, hence the standard error is 37.093, and the price of the (next) diamond ring of 0.43 carat is at least 1265.75 and at most 1415.28 dollar. The ci.reg function sets the type of n for single value prediction. 𝑠 𝑝𝑟𝑒𝑑 2 = 𝑠 𝑌 ℎ 2 + 𝑠 2 = 362.14+1013.78=1375.92 Yˆh ± 𝑡 𝑐 𝑠 𝑝𝑟𝑒𝑑 = 1340.415±2.013 37.093 =1265.75, 1415.08

The Diamond example, if 𝑋 ℎ =0.43 , compute 3. The confidence interval for the mean price 𝑌 ℎ 𝑛𝑒𝑤 of three diamonds with the same weight (0.43) Where: 𝑠 2 𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛 = 𝑀𝑆𝐸 𝑚 + 𝑠 2 𝑌 ℎ =𝑀𝑆𝐸[ 1 𝑚 + 1 𝑛 + 𝑋 ℎ − 𝑋 2 Σ 𝑋 𝑖 − 𝑋 2 ] 𝑌 ℎ ± 𝑡 𝑐 𝑠{𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛} The last example in this topic is to make prediction of the mean of next three diamonds, all has the weight of 0.43 carat. The variance of the mean of 3 prediction can be calculated as 700.07 as shown here. The standard error is 26.46, this is value that is smaller than single value prediction but bigger than the mean prediction. The average mean of 3 diamond ring is at least 1287 and at most 1394 dollars. And to compute this in R, set the ci.reg function with type = nm, and m=3. 𝑠 2 𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛 = 𝑀𝑆𝐸 𝑚 + 𝑠 2 𝑌 ℎ = 𝑌 ℎ ± 𝑡 𝑐 𝑠{𝑝𝑟𝑒𝑑𝑚𝑒𝑎𝑛}=1340.415 ±2.013 26.46 =(1287.151, 1393.679) 31.84 2 3 +362.14 = 700.07