Exploratory Data Analysis

John Tukey Developed these procedures to help one get a first look at distributions of scores. What is the shape of the distribution? Are there any suspicious scores. Stem and Leaf Display Box and Whiskers Plot

Stem and Leaf Display See the pulse rate data at Exploratory Data Analysis (EDA).Exploratory Data Analysis (EDA) The scores range from 48 to 104. We probably want to group them into 5 to 15 intervals. I’ll use two intervals for the 40’s, two for the 50’s, etc.

The Stem Consists of a column of leading (aka “most significant” digits, the leftmost digits in the scores. I’ll add to the stem the leaves, the trailing (rightmost, least significant) digits of each score

The Stem With Leaves Next, I’ll arrange the leaves (within each row) from lowest to highest and add a “depth” column.

Leaves Arranged in Order

The Depth Column This column tells you how many scores there are in that row and all rows between it and the closer tail of the distribution. The row that contains the median has the row frequency in parentheses.

Rotated Display It looks like a histogram, but the bars made up of the scores. From this display, can you identify any scores that are odd, compared to most of the other scores?

Box and Whisker Plot Median Location = (N + 1)/2 = 97/2 =48.5. The median will be located between the 48 th and the 49 th scores from either tail.

Are 40 scores from 68 to 48. Count up 8 more scores, starting with the first 70. The 48 th score is a 70, the 49 th score is a 70, the median is 70.

The Hinge Location = (Median Location + 1)/2. Drop any decimal on the median location For our data, hinge location = (48 + 1)/2 = Now, the upper hinge is the 24.5 th score from the upper end of the distribution.

There are 24 scores from 80 up to 104. Go in toward the median one more score. The 25 th score from the highest is a 78. The upper hinge is ( )/2 = 79.

The 26 th score from the lowest score is a 64. Move towards the lower tail by one score and you see the 25 th score is also a 64. One more, the 24 th score is also a 64. The lower hinge is 64.

The H-Spread = the difference between the upper hinge and the lower hinge. For our data, = 15. This is the range of the middle 50% of the scores. You also know this as the interquartile range.

The Inner Fences The upper inner fence = the upper hinge plus 1.5 H ‑ spreads. For our data, (15) = The lower inner fence is the lower hinge minus 1.5 H ‑ spreads, (15) = These are invisible fences, they are not plotted.

Adjacent Values These are scores that are outside of the middle 50% of the scores but within the inner fences. For our data, these will be scores that fall –between 79 and or –between 41.5 and 64

Outliers These are scores that are beyond the inner fences. For our data, these are scores that are –Less than 41.5 or –Greater than 101.5

Outer Fences These invisible fences are 3 H-spreads beyond the hinges. For our data the lower outer fence is at (15) = 34 and the upper outer fence is at (15) = 124 Scores that are beyond the outer fences are called way-outliers.

Drawing the Plot Prepare a numerical scale. Draw a box that extends from the lower hinge to the upper hinge. Draw a line through the box at the median. May also insert a symbol at the mean. Draw whiskers out to the most extreme adjacent values on each side

Whiskers For our data, the lowest adjacent value is the 48, so we draw the whiskers on the lower end out to 48. We do not go all the way out the inner fence unless there is a score there. The highest adjacent value is a 99, so we draw whiskers on the upper end out to 99.

Outliers Every outlier is plotted with a special symbol, often a O for a regular outlier and an * for a way-outliers. Some programs will also print the identification number next to every outlier These days, we use statistical software to make these displays and plots rather than doing them by hand.

Plots Produced by SAS

How tall, in inches, is your ideal mate?

Eight Foot Tall Mate ! That is a WAY-OUTLIER for sure ! Investigation of the original data sheets revealed that the actual response was 69 inches, not 96 inches.

Exploratory Data Analysis (EDA) It is highly recommended that you read the document linked above. It includes additional examples and a bit of silliness that might help you remember key concepts. Do watch the video clip of the Id attempting to cross an outer fence on the Forbidden Planet.