Data Mining: EXPLORING DATA Instructor: Dr. Chun Yu School of Statistics Jiangxi University of Finance and Economics Fall 2016

What is data exploration? A preliminary exploration of the data to better understand its characteristics. In our discussion of data exploration, we focus on Summary statistics Visualization Online Analytical Processing (OLAP)

Summary Statistics Summary statistics are numbers that summarize properties of the data Summarized properties include frequency, location and spread Examples: location - mean spread - standard deviation Most summary statistics can be calculated in a single pass through the data

Frequency  

Mode The mode of an attribute is the most frequent attribute value The mode of the class attribute is freshmen, with a frequency of 0.33 The notions of frequency and mode are typically used with categorical data Class Size Frequency Freshmen Sophomore Junior Senior Total 200 160 130 110 600 200/600 = 0.33 160/600 = 0.27 130/600 = 0.22 110/600 = 0.18 1.00

Measures of Location: Mean and Median The mean and the median are the most common measures of the location of a set of points.

Median A sample median is the middle sorted observation. That is, we want a value such that half of the data is below it and half above it. How to calculate the median? Step 1: Sort the data from smallest to largest. Step 2: If n is odd, pick the middle observation. If n is even, average the two middle observations.

Mean and Median Example 1: Computation of the median with an odd number of data points. The data: 7, 11, 7, 14, 13 The data put in order: 7, 7, 11, 13, 14 Median: 11 Mean: (7 + 7 + 11 + 13 + 14)/5 = 10.4

Mean and Median Example 2: Computation of the median with an even number of data points. The data: 7, 11, 7, 14, 13, 15 The data put in order: 7, 7, 11, 13, 14, 15 Median: 12, the average of 11 and 13 Mean: (7 + 7 + 11 + 13 + 14 + 15)/6 = 11.17

Effect of Outlier on Mean and Median Begin with data 7, 7, 11, 13, 14 as in Example 1. What happens to the mean and median when the largest value is changed from 14 to 140? Change affects the mean but not the median. Median is still 11 but mean is 35.6. The mean “chases after” extreme observations.

Mean and Median When the data are symmetric, the median and mean will be about the same. When the data are skewed right, the mean is greater than the median. (Ex: Income) When the data are skewed left, the mean is less than the median. (Ex: Exam scores.)

Percentiles For continuous data, the notion of a percentile is more useful. Given an ordinal or continuous attribute x and a number p between 0 and 100, the pth percentile is a value xp of x such that p% of the observed values of x are less than xp . For instance, the 50th percentile is the value x50 such that 50% of all values of x are less than x50 . The median is the 50th percentile.

Calculating the pth Percentile Data: X1, X2, X3, …, Xn 1. Sort the data from smallest to largest. 2. Compute the index: i =(p/100)*n 3. If i is: (a) an integer. Find the ith observation in the ordered data and the (i+1)th observation. The average of these two is the pth percentile. (b) not an integer, round UP to the next largest integer. This observation in the ordered data is the pth percentile.

Calculating the pth Percentile Example Sorted heights (cm): 165, 165, 167, 168,170,172,173,175, 180, 190. What is the 50th percentile? Compute the index: i = (50/100)*10 = 5 The 50th percentile is the average of 5th and 6th observations The 50th percentile is: (170+172)/2 = 171

Quartiles Divide the data into four groups Q1 = first quartile = 25th percentile Q2 = second quartile = 50th percentile = median Q3 = third quartile = 75th percentile In the previous example, What is the first quartile? What is the median (second quartile)? What is the third quartile?

Quartiles 1. First quartile, Q1 i = (25/100)*10 = 2.5 Q1 is the third observation, that is, Q1 = 167 2. Median, Q2 = 171 3. Third quartile, Q3 i = (75/100)*10 = 7.5 Q3 is the 8th observation, that is, Q3 = 175

Measures of Spread: Range and Variance  

Standard Deviation Data: 1, 2, 3, 4, 5 What is the standard deviation of 6,7,8,9,10? What is the standard deviation of -1, -2, -3, -4, -5? What is the standard deviation of 5, 10, 15, 20, 25? xi 1 -2 4 4+1+0+1+4 = 10 10/4 = 2.5 Sqrt(2.5) = 1.58 2 -1 3 5

Visualization Visualization is the conversion of data into a visual or tabular format so that the characteristics of the data and the relationships among data items or attributes can be analyzed or reported. Visualization of data is one of the most powerful and appealing techniques for data exploration. Humans have a well developed ability to analyze large amounts of information that is presented visually Can detect general patterns and trends Can detect outliers and unusual patterns

Visualization Techniques: Histograms Usually shows the distribution of values of a single variable Divide the values into bins and show a bar plot of the number of objects in each bin. The height of each bar indicates the number of objects Shape of histogram depends on the number of bins Example: Petal Width (10 and 20 bins, respectively)

Two-Dimensional Histograms Show the joint distribution of the values of two attributes Example: petal width and petal length What does this tell us?

Stem and Leaf Plot Each number is broken into a stem and a leaf such that the last digit is leaf and all other leading digits are a stem Place the stems in increasing order to the left of a vertical line To the right of the vertical line, place the leaves in ascending order Exam scores for n = 30 students: 41 46 47 58 59 67 68 70 70 70 74 75 77 77 78 80 81 82 82 83 84 84 85 85 86 87 92 94 96 97

Stem and Leaf Plot 4| 1 6 7 5| 8 9 6| 7 8 7| 0 0 0 4 5 5 7 8 8| 0 1 2 2 3 4 4 5 5 6 7 9| 2 4 6 7 Advantages: shows the actual data values Shows the rank order of the data Shows the shape of the data set Easy and quick to do by hand for small data sets

Visualization Techniques: Box Plots Invented by J. Tukey Another way of displaying the distribution of data Following figure shows the basic part of a box plot outlier 10th percentile 25th percentile 75th percentile 50th percentile

Example of Box Plots Box plots can be used to compare attributes

Pie Chart Present the percent frequency distribution Draw a circle Divide the circle into pieces that correspond to the percent frequency distribution

Visualization Techniques: Scatter Plots Attributes values determine the position Two-dimensional scatter plots most common, but can have three-dimensional scatter plots Often additional attributes can be displayed by using the size, shape, and color of the markers that represent the objects It is useful to have arrays of scatter plots can compactly summarize the relationships of several pairs of attributes See example on the next slide

Scatter Plot Array of Iris Attributes

OLAP On-Line Analytical Processing (OLAP) was proposed by E. F. Codd, the father of the relational database. Relational databases put data into tables, while OLAP uses a multidimensional array representation. Such representations of data previously existed in statistics and other fields There are a number of data analysis and data exploration operations that are easier with such a data representation.

Data Warehouses and Example A data warehouse is usually modeled by a multidimensional data structure, called a data cube Example: A data cube for a company The cube has three dimensions: address (with city values Chicago, New York, Toronto, Vancouver) time (with quarter values Q1, Q2, Q3, Q4) item(with item type values home entertainment, computer, phone, security)

A multidimensional data cube

A multidimensional data cube: drill down and roll up

Thank you!