The Examination of Residuals. The residuals are defined as the n differences : where is an observation and is the corresponding fitted value obtained.

Slides:



Advertisements
Similar presentations
Assumptions underlying regression analysis
Advertisements

Hypothesis Testing. To define a statistical Test we 1.Choose a statistic (called the test statistic) 2.Divide the range of possible values for the test.
11-1 Empirical Models Many problems in engineering and science involve exploring the relationships between two or more variables. Regression analysis.
Incomplete Block Designs. Randomized Block Design We want to compare t treatments Group the N = bt experimental units into b homogeneous blocks of size.
1 Chapter 4 Experiments with Blocking Factors The Randomized Complete Block Design Nuisance factor: a design factor that probably has an effect.
Chapter 4 Randomized Blocks, Latin Squares, and Related Designs
Inference for Regression
6-1 Introduction To Empirical Models 6-1 Introduction To Empirical Models.
11 Simple Linear Regression and Correlation CHAPTER OUTLINE
Regression Analysis Once a linear relationship is defined, the independent variable can be used to forecast the dependent variable. Y ^ = bo + bX bo is.
Copyright (c) 2004 Brooks/Cole, a division of Thomson Learning, Inc. Chapter 13 Nonlinear and Multiple Regression.
Comparing k Populations Means – One way Analysis of Variance (ANOVA)
Chapter 13 Additional Topics in Regression Analysis
BA 555 Practical Business Analysis
Additional Topics in Regression Analysis
Copyright (c) 2004 Brooks/Cole, a division of Thomson Learning, Inc. Chapter 14 Goodness-of-Fit Tests and Categorical Data Analysis.
Inferences About Process Quality
Analysis of Variance & Multivariate Analysis of Variance
11-1 Empirical Models Many problems in engineering and science involve exploring the relationships between two or more variables. Regression analysis.
1 1 Slide © 2008 Thomson South-Western. All Rights Reserved Slides by JOHN LOUCKS & Updated by SPIROS VELIANITIS.
Regression Analysis Regression analysis is a statistical technique that is very useful for exploring the relationships between two or more variables (one.
Inference for regression - Simple linear regression
Regression Analysis (2)
Copyright © 2013, 2010 and 2007 Pearson Education, Inc. Chapter Inference on the Least-Squares Regression Model and Multiple Regression 14.
The paired sample experiment The paired t test. Frequently one is interested in comparing the effects of two treatments (drugs, etc…) on a response variable.
The Examination of Residuals. The residuals are defined as the n differences : where is an observation and is the corresponding fitted value obtained.
Non-Linear Models. Non-Linear Growth models many models cannot be transformed into a linear model The Mechanistic Growth Model Equation: or (ignoring.
1 1 Slide © 2005 Thomson/South-Western Slides Prepared by JOHN S. LOUCKS St. Edward’s University Slides Prepared by JOHN S. LOUCKS St. Edward’s University.
1 1 Slide Simple Linear Regression Part A n Simple Linear Regression Model n Least Squares Method n Coefficient of Determination n Model Assumptions n.
MANOVA Multivariate Analysis of Variance. One way Analysis of Variance (ANOVA) Comparing k Populations.
Applications The General Linear Model. Transformations.
1 1 Slide © 2004 Thomson/South-Western Slides Prepared by JOHN S. LOUCKS St. Edward’s University Slides Prepared by JOHN S. LOUCKS St. Edward’s University.
Chapter 11: Applications of Chi-Square. Count or Frequency Data Many problems for which the data is categorized and the results shown by way of counts.
Statistical Decision Making. Almost all problems in statistics can be formulated as a problem of making a decision. That is given some data observed from.
Chapter 11 Multifactor Analysis of Variance.
1 1 Slide © 2014 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole.
The Examination of Residuals. Examination of Residuals The fitting of models to data is done using an iterative approach. The first step is to fit a simple.
Chap 14-1 Statistics for Business and Economics, 6e © 2007 Pearson Education, Inc. Chapter 14 Additional Topics in Regression Analysis Statistics for Business.
MANOVA Multivariate Analysis of Variance. One way Analysis of Variance (ANOVA) Comparing k Populations.
Chapter 4 Linear Regression 1. Introduction Managerial decisions are often based on the relationship between two or more variables. For example, after.
1 11 Simple Linear Regression and Correlation 11-1 Empirical Models 11-2 Simple Linear Regression 11-3 Properties of the Least Squares Estimators 11-4.
Orthogonal Linear Contrasts This is a technique for partitioning ANOVA sum of squares into individual degrees of freedom.
Comparing k Populations Means – One way Analysis of Variance (ANOVA)
1 1 Slide © 2003 South-Western/Thomson Learning™ Slides Prepared by JOHN S. LOUCKS St. Edward’s University.
Two-Way (Independent) ANOVA. PSYC 6130A, PROF. J. ELDER 2 Two-Way ANOVA “Two-Way” means groups are defined by 2 independent variables. These IVs are typically.
Correlation & Regression Analysis
ANOVA TABLE Factorial Experiment Completely Randomized Design.
The p-value approach to Hypothesis Testing
Factorial Experiments Analysis of Variance Experimental Design.
Assumptions of Multiple Regression 1. Form of Relationship: –linear vs nonlinear –Main effects vs interaction effects 2. All relevant variables present.
Chapter 11: Categorical Data n Chi-square goodness of fit test allows us to examine a single distribution of a categorical variable in a population. n.
1 Chapter 5.8 What if We Have More Than Two Samples?
Experimental Designs The objective of Experimental design is to reduce the magnitude of random error resulting in more powerful tests to detect experimental.
Statistical Decision Making. Almost all problems in statistics can be formulated as a problem of making a decision. That is given some data observed from.
Comparing k Populations Means – One way Analysis of Variance (ANOVA)
Chapter 4 Basic Estimation Techniques
Inference for Least Squares Lines
11-1 Empirical Models Many problems in engineering and science involve exploring the relationships between two or more variables. Regression analysis.
Slides by JOHN LOUCKS St. Edward’s University.
Comparing k Populations
Prepared by Lee Revere and John Large
Non-Linear Regression
Hypothesis testing and Estimation
Factorial Experiments
Product moment correlation
ANOVA Analysis of Variance.
The Examination of Residuals
Diagnostics and Remedial Measures
Diagnostics and Remedial Measures
One way Analysis of Variance (ANOVA)
Presentation transcript:

The Examination of Residuals

The residuals are defined as the n differences : where is an observation and is the corresponding fitted value obtained by use of the fitted model.

Many of the statistical procedures used in linear and nonlinear regression analysis are based certain assumptions about the random departures from the proposed model. Namely; the random departures are assumed i) to have zero mean, ii) to have a constant variance,  2, iii) independent, and iv) follow a normal distribution.

Thus if the fitted model is correct, the residuals should exhibit tendencies that tend to confirm the above assumptions, or at least, should not exhibit a denial of the assumptions.

The principal ways of plotting the residuals e i are: 1. Overall. 3. Against the fitted values 2. In time sequence, if the order is known. 4. Against the independent variables x ij for each value of j In addition to these basic plots, the residuals should also be plotted 5. In any way that is sensible for the particular problem under consideration,

Overall Plot The residuals can be plotted in an overall plot in several ways.

1.The scatter plot. 2.The histogram. 3.The box-whisker plot. 4.The kernel density plot 5.a normal plot or a half normal plot on standard probability paper.

2.The Chi-square goodness of fit test The standard statistical test for testing Normality are: 1.The Kolmogorov-Smirnov test.

The empirical distribution function is defined below for n random observations The Kolmogorov-Smirnov test The Kolmogorov-Smirnov uses the empirical cumulative distribution function as a tool for testing the goodness of fit of a distribution. F n (x) = the proportion of observations in the sample that are less than or equal to x.

Let F 0 (x) denote the hypothesized cumulative distribution function of the population (Normal population if we were testing normality) If F 0 (x) truly represented distribution of observations in the population than F n (x) will be close to F 0 (x) for all values of x.

The Kolmogorov-Smirinov test statistic is : = the maximum distance between F n (x) and F 0 (x). If F 0 (x) does not provide a good fit to the distributions of the observation - D n will be large. Critical values for are given in many texts

Let f i denote the observed frequency in each of the class intervals of the histogram. The Chi-square goodness of fit test The Chi-square test uses the histogram as a tool for testing the goodness of fit of a distribution. Let E i denote the expected number of observation in each class interval assuming the hypothesized distribution.

m = the number of class intervals used for constructing the histogram). The hypothesized distribution is rejected if the statistic: is large. (greater than the critical value from the chi-square distribution with m - 1 degrees of freedom.

Note. The in the above tests it is assumed that the residuals are independent with a common variance of  2. This is not completely accurate for this reason: Although the theoretical random errors  i are all assumed to be independent with the same variance  2, the residuals are not independent and they also do not have the same variance.

They will however be approximately independent with common variance if the sample size is large relative to the number of parameters in the model. It is important to keep this in mind when judging residuals when the number of observations is close to the number of parameters in the model.

Time Sequence Plot The residuals should exhibit a pattern of independence. If the data was collected in time there could be a strong possibility that the random departures from the model are autocorrelated.

Namely the random departures for observations that were taken at neighbouring points in time are autocorrelated. This autocorrelation can sometimes be seen in a time sequence plot. The following three graphs show a sequence of residuals that are respectively i) positively autocorrelated, ii) independent and iii) negatively autocorrelated.

i) Positively auto-correlated residuals

ii) Independent residuals

iii) Negatively auto-correlated residuals

There are several statistics and statistical tests that can also pick out autocorrelation amongst the residuals. The most common are: ii)The autocorrelation function i)The Durbin Watson statistic iii)The runs test

The Durbin Watson statistic : If the residuals are serially correlated the differences, e i - e i+1, will be stochastically small. Hence a small value of the Durbin-Watson statistic will indicate positive autocorrelation. Large values of the Durbin-Watson statistic on the other hand will indicate negative autocorrelation. Critical values for this statistic, can be found in many statistical textbooks. The Durbin-Watson statistic which is used frequently to detect serial correlation is defined by the following formula:

The autocorrelation function: This statistic measures the correlation between residuals the occur a distance k apart in time. One would expect that residuals that are close in time are more correlated than residuals that are separated by a greater distance in time. If the residuals are independent than r k should be close to zero for all values of k A plot of r k versus k can be very revealing with respect to the independence of the residuals. Some typical patterns of the autocorrelation function are given below: The autocorrelation function at lag k is defined by :

This statistic measures the correlation between residuals the occur a distance k apart in time. One would expect that residuals that are close in time are more correlated than residuals that are separated by a greater distance in time. If the residuals are independent than r k should be close to zero for all values of k A plot of r k versus k can be very revealing with respect to the independence of the residuals.

Some typical patterns of the autocorrelation function are given below: Auto correlation pattern for independent residuals

Various Autocorrelation patterns for serially correlated residuals

The runs test: This test uses the fact that the residuals will oscillate about zero at a “normal” rate if the random departures are independent. If the residuals oscillate slowly about zero, this is an indication that there is a positive autocorrelation amongst the residuals. If the residuals oscillate at a frequent rate about zero, this is an indication that there is a negative autocorrelation amongst the residuals.

In the “runs test”, one observes the time sequence of the “sign” of the residuals: and counts the number of runs (i.e. the number of periods that the residuals keep the same sign). This should be low if the residuals are positively correlated and high if negatively correlated.

Plot Against fitted values and the Predictor Variables X ij If we "step back" from this diagram and the residuals behave in a manner consistent with the assumptions of the model we obtain the impression of a horizontal "band " of residuals which can be represented by the diagram below.

Individual observations lying considerably outside of this band indicate that the observation may be and outlier. An outlier is an observation that is not following the normal pattern of the other observations. Such an observation can have a considerable effect on the estimation of the parameters of a model. Sometimes the outlier has occurred because of a typographical error. If this is the case and it is detected than a correction can be made. If the outlier occurs for other (and more natural) reasons it may be appropriate to construct a model that incorporates the occurrence of outliers.

If our "step back" view of the residuals resembled any of those shown below we should conclude that assumptions about the model are incorrect. Each pattern may indicate that a different assumption may have to be made to explain the “abnormal” residual pattern. b) a)

Pattern a) indicates that the variance the random departures is not constant (homogeneous) but increases as the value along the horizontal axis increases (time, or one of the independent variables). This indicates that a weighted least squares analysis should be used. The second pattern, b) indicates that the mean value of the residuals is not zero. Linear and quadratic terms have been omitted that should have been included in the model. This is usually because the model (linear or non linear) has not been correctly specified.

Example – Analysis of Residuals Motor Vehicle Data Dependent = mpg Independent = Engine size, horsepower and weight

When a linear model was fit and residuals examined graphically the following plot resulted:

The pattern that we are looking for is:

The pattern that was found is: This indicates a nonlinear relationship: This can be handle by adding polynomial terms (quadratic, cubic, quartic etc.) of the independent variables or transforming the dependent variable

Performing the log transformation on the dependent variable (mpg) results in the following residual plot There still remains some non linearity

The log transformation

The Box-Cox transformations = 2 = 0 = -1 = 1 = -1

The log ( = 0) transformation was not totally successful - try moving further down the staircase of the family of transformations ( = -0.5)

try moving a bit further down the staircase of the family of transformations ( = -1.0)

The results after deleting the outlier are given below:

This corresponds to the model or and

Checking normality with a P-P plot

Example Non-Linear Regression

In this example we are measuring the amount of a compound in the soil: 1.7 days after application 2.14 days after application 3.21 days after application 4.28 days after application 5.42 days after application 6.56 days after application 7.70 days after application 8.84 days after application

This is carried out at two test plot locations 1.Craik 2.Tilson 6 measurements per location are made each time

The data

Graph

The Model: Exponential decay with nonzero asymptote c a

Some starting values of the parameters found by trial and error by Excel

Non Linear least squares iteration by SPSS (Craik)

ANOVA Table (Craik) Parameter Estimates (Craik)

Testing Hypothesis: similar to linear regression Caution: This statistic has only an approximate F – distribution when the sample size is large

Example: Suppose we want to test H 0 : c = 0 against H A : c ≠ 0 Complete model Reduced model

ANOVA Table (Complete model) ANOVA Table (Reduced model)

The Test

Use of Dummy Variables Non Linear Regression

The Model: or where

The data file

Non Linear least squares iteration by SPSS

ANOVA Table Parameter Estimates

Testing Hypothesis: Suppose we want to test H 0 :  a = a 1 – a 2 = 0 and  k = k 1 – k 2 = 0

The Reduced Model: or

ANOVA Table Parameter Estimates

The F Test Thus we accept the null Hypothesis that the reduced model is correct

Factorial Experiments Analysis of Variance Experimental Design

Dependent variable Y k Categorical independent variables A, B, C, … (the Factors) Let –a = the number of categories of A –b = the number of categories of B –c = the number of categories of C –etc.

The Completely Randomized Design We form the set of all treatment combinations – the set of all combinations of the k factors Total number of treatment combinations –t = abc…. In the completely randomized design n experimental units (test animals, test plots, etc. are randomly assigned to each treatment combination. –Total number of experimental units N = nt=nabc..

The treatment combinations can thought to be arranged in a k-dimensional rectangular block A 1 2 a B 12b

A B C

The Completely Randomized Design is called balanced If the number of observations per treatment combination is unequal the design is called unbalanced. (resulting mathematically more complex analysis and computations) If for some of the treatment combinations there are no observations the design is called incomplete. (some of the parameters - main effects and interactions - cannot be estimated.)

Example In this example we are examining the effect of We have n = 10 test animals randomly assigned to k = 6 diets tThe level of protein A (High or Low) and tThe source of protein B (Beef, Cereal, or Pork) on weight gains (grams) in rats.

The k = 6 diets are the 6 = 3×2 Level-Source combinations 1.High - Beef 2.High - Cereal 3.High - Pork 4.Low - Beef 5.Low - Cereal 6.Low - Pork

Table Gains in weight (grams) for rats under six diets differing in level of protein (High or Low) and s ource of protein (Beef, Cereal, or Pork) Level of ProteinHigh ProteinLow protein Source of ProteinBeefCerealPorkBeefCerealPork Diet Mean Std. Dev

Example – Four factor experiment Four factors are studied for their effect on Y (luster of paint film). The four factors are: Two observations of film luster (Y) are taken for each treatment combination 1) Film Thickness - (1 or 2 mils) 2)Drying conditions (Regular or Special) 3)Length of wash (10,30,40 or 60 Minutes), and 4)Temperature of wash (92 ˚C or 100 ˚C)

The data is tabulated below: Regular DrySpecial Dry Minutes92  C100  C92  C100  C 1-mil Thickness mil Thickness

Notation Let the single observations be denoted by a single letter and a number of subscripts y ijk…..l The number of subscripts is equal to: (the number of factors) st subscript = level of first factor 2 nd subscript = level of 2 nd factor … Last subsrcript denotes different observations on the same treatment combination

Notation for Means When averaging over one or several subscripts we put a “bar” above the letter and replace the subscripts by  Example: y 241   

Profile of a Factor Plot of observations means vs. levels of the factor. The levels of the other factors may be held constant or we may average over the other levels

Level of ProteinBeefCerealPorkOverall Low Source of Protein High Overall Summary Table

Profiles of Weight Gain for Source and Level of Protein

Effects in a factorial Experiment

Mean

Main Effects for Factor A (Source of Protein) BeefCerealPork

Main Effects for Factor B (Level of Protein) HighLow

AB Interaction Effects Source of Protein BeefCerealPork LevelHigh of Protein Low

Example 2 Paint Luster Experiment

Table: Means and Cell Frequencies

Means and Frequencies for the AB Interaction (Temp - Drying)

Profiles showing Temp-Dry Interaction

Means and Frequencies for the AD Interaction (Temp- Thickness)

Profiles showing Temp-Thickness Interaction

The Main Effect of C (Length)

Additive Factors A B

Interacting Factors A B

Models for factorial Experiments Single Factor: y ij =  +  i +  ij i = 1,2,...,a; j = 1,2,...,n Two Factor: y ijk =  +  i +  j +  ij +  ijk i = 1,2,...,a ; j = 1,2,...,b ; k = 1,2,...,n

Three Factor: y ijkl =  +  i +  j +  ij +  k + (  ik + (  jk +  ijk +  ijkl =  +  i +  j +  k +  ij + (  ik + (  jk +  ijk +  ijkl i = 1,2,...,a ; j = 1,2,...,b ; k = 1,2,...,c; l = 1,2,...,n

Four Factor: y ijklm =  +  i +  j +  ij +  k + (  ik + (  jk +  ijk +  l + (  il + (  jl +  ijl + (  kl + (  ikl + (  jkl +  ijkl +  ijklm =  +  i +  j +  k +  l +  ij + (  ik + (  jk + (  il + (  jl + (  kl +  ijk +  ijl + (  ikl + (  jkl +  ijkl +  ijklm i = 1,2,...,a ; j = 1,2,...,b ; k = 1,2,...,c; l = 1,2,...,d; m = 1,2,...,n where0 =   i =   j =   ij =   k =  (  ik =  (  jk =   ijk =   l =  (  il =  (  jl =   ijl =  (  kl =  (  ikl =  (  jkl =   ijkl and  denotes the summation over any of the subscripts.

Estimation of Main Effects and Interactions Estimator of Main effect of a Factor Estimator of k-factor interaction effect at a combination of levels of the k factors = Mean at the combination of levels of the k factors - sum of all means at k-1 combinations of levels of the k factors +sum of all means at k-2 combinations of levels of the k factors - etc. =Mean at level i of the factor - Overall Mean

Example: The main effect of factor B at level j in a four factor (A,B,C and D) experiment is estimated by: The two-factor interaction effect between factors B and C when B is at level j and C is at level k is estimated by:

The three-factor interaction effect between factors B, C and D when B is at level j, C is at level k and D is at level l is estimated by: Finally the four-factor interaction effect between factors A,B, C and when A is at level i, B is at level j, C is at level k and D is at level l is estimated by:

Definition: A factor is said to not affect the response if the profile of the factor is horizontal for all combinations of levels of the other factors: No change in the response when you change the levels of the factor (true for all combinations of levels of the other factors) Otherwise the factor is said to affect the response:

Definition: Two (or more) factors are said to interact if changes in the response when you change the level of one factor depend on the level(s) of the other factor(s). Profiles of the factor for different levels of the other factor(s) are not parallel Otherwise the factors are said to be additive. Profiles of the factor for different levels of the other factor(s) are parallel.

If two (or more) factors interact each factor effects the response. If two (or more) factors are additive it still remains to be determined if the factors affect the response In factorial experiments we are interested in determining –which factors effect the response and – which groups of factors interact.

The testing in factorial experiments 1.Test first the higher order interactions. 2.If an interaction is present there is no need to test lower order interactions or main effects involving those factors. All factors in the interaction affect the response and they interact 3.The testing continues with for lower order interactions and main effects for factors which have not yet been determined to affect the response.

Models for factorial Experiments Single Factor: y ij =  +  i +  ij i = 1,2,...,a; j = 1,2,...,n Two Factor: y ijk =  +  i +  j +  ij +  ijk i = 1,2,...,a ; j = 1,2,...,b ; k = 1,2,...,n

Three Factor: y ijkl =  +  i +  j +  ij +  k + (  ik + (  jk +  ijk +  ijkl =  +  i +  j +  k +  ij + (  ik + (  jk +  ijk +  ijkl i = 1,2,...,a ; j = 1,2,...,b ; k = 1,2,...,c; l = 1,2,...,n

Four Factor: y ijklm =  +  i +  j +  ij +  k + (  ik + (  jk +  ijk +  l + (  il + (  jl +  ijl + (  kl + (  ikl + (  jkl +  ijkl +  ijklm =  +  i +  j +  k +  l +  ij + (  ik + (  jk + (  il + (  jl + (  kl +  ijk +  ijl + (  ikl + (  jkl +  ijkl +  ijklm i = 1,2,...,a ; j = 1,2,...,b ; k = 1,2,...,c; l = 1,2,...,d; m = 1,2,...,n where0 =   i =   j =   ij =   k =  (  ik =  (  jk =   ijk =   l =  (  il =  (  jl =   ijl =  (  kl =  (  ikl =  (  jkl =   ijkl and  denotes the summation over any of the subscripts.

Estimation of Main Effects and Interactions Estimator of Main effect of a Factor Estimator of k-factor interaction effect at a combination of levels of the k factors = Mean at the combination of levels of the k factors - sum of all means at k-1 combinations of levels of the k factors +sum of all means at k-2 combinations of levels of the k factors - etc. =Mean at level i of the factor - Overall Mean

Example: The main effect of factor B at level j in a four factor (A,B,C and D) experiment is estimated by: The two-factor interaction effect between factors B and C when B is at level j and C is at level k is estimated by:

The three-factor interaction effect between factors B, C and D when B is at level j, C is at level k and D is at level l is estimated by: Finally the four-factor interaction effect between factors A,B, C and when A is at level i, B is at level j, C is at level k and D is at level l is estimated by:

Anova Table entries Sum of squares interaction (or main) effects being tested  (product of sample size and levels of factors not included in the interaction) Degrees of freedom = df = product of (number of levels - 1) of factors included in the interaction.

Mean

Main Effects for Factor A (Source of Protein) BeefCerealPork

Main Effects for Factor B (Level of Protein) HighLow

AB Interaction Effects Source of Protein BeefCerealPork LevelHigh of Protein Low

Table: Means and Cell Frequencies

Means and Frequencies for the AB Interaction (Temp - Drying)

Profiles showing Temp-Dry Interaction

Means and Frequencies for the AD Interaction (Temp- Thickness)

Profiles showing Temp-Thickness Interaction

The Main Effect of C (Length)