1. Data mining methodology in Weka
Renata Benda Prokeinova
Department of Statistics and Operation research

2. Logistic regression- teory
Logistic regression can in many ways be seen to be similar to ordinary regression.
It models the relationship between a dependent and one or more independent variables, and allows us to look at the fit of the model as well as at the significance of the relationships (between dependent and independent variables) that we are modelling. However, the underlying principle of binomial logistic regression, and its statistical calculation, are quite different to ordinary linear regression.
While ordinary regression uses ordinary least squares to find a best fitting line, and comes up with coefficients that predict the change in the dependent variable for one unit change in the independent variable, logistic regression estimates the probability of an event occurring.
What we want to predict from a knowledge of relevant independent variables is not a precise numerical value of a dependent variable, but rather the probability (p) that it is 1 (event occurring) rather than 0 (event not occurring).
This means that, while in linear regression, the relationship between the dependent and the independent variables is linear, this assumption is not made in logistic regression.

3. Logistic regression -output
First section of the report: Coefficients... Class Variable yes =============================== outlook=sunny outlook=overcast outlook=rainy temperature humidity windy Intercept The coefficients are in fact the weights that are applied to each attribute before adding them together. However, the result is the probability that the new instance belongs to class yes (> 0.5 means yes).

4. Logistic regression -output
Odds Ratios... Class Variable yes =============================== outlook=sunny outlook=overcast outlook=rainy temperature humidity windy The odds ratios indicate how large of an influence a change in that value (or change to that value) will have on the prediction. I think this link does a great job explaining the odds ratios. The value of outlook=overcast is so large because if the outlook is overcast the odds are very good that play will equal yes.

5. Logistic regression -output
=== Confusion Matrix === a b <-- classified as 7 2 | a = yes 1 4 | b = no The confusion matrix simply shows you how many of the test data points are correctly and incorrectly classified. In your example 7 A's were actually classified as A, where as 2 A's were misclassified as B. Your question is more thoroughly answered in this question: How to read the classifier confusion matrix in WEKA.

6. Datasets

7. Market basket analysis
Market Basket Analysis is a modelling technique based upon the theory that if you buy a certain group of items, you are more (or less) likely to buy another group of items. For example, if you are in an English pub and you buy a pint of beer and don't buy a bar meal, you are more likely to buy crisps (US. chips) at the same time than somebody who didn't buy beer.
association-rule-learning/

8. Market basket analysis: the basics terminology
Items are the objects that we are identifying associations between.
Transactions are instances of groups of items co-occuring together.
The support of an item or item set is the fraction of transactions in our data set that contain that item or item set. In general, it is nice to identify rules that have a high support, as these will be applicable to a large number of transactions. For super market retailers, this is likely to involve basic products that are popular across an entire user base (e.g. bread, milk). A printer cartridge retailer, for example, may not have products with a high support, because each customer only buys cartridges that are specific to his / her own printer.

9. Market basket analysis: the basics terminology
The confidence of a rule is the likelihood that it is true for a new transaction that contains the items on the LHS of the rule:
The lift of a rule is the ratio of the support of the items on the LHS of the rule co-occuring with items on the RHS divided by probability that the LHS and RHS co-occur if the two are independent
confidence(im⇒in)= support(im∪in)/ support(im)
lift(im⇒in)= support(im∪in)/(support(im)×support(in))

10. Factor and Cluster Analysis
Chapter Twenty-one
Factor and Cluster Analysis

11. Factor Analysis Combines questions or variables to create new factors
Combines objects to create new groups
Uses in Data Analysis
To identify underlying constructs in the data from the groupings of variables that emerge
To reduce the number of variables to a more manageable set

12. Factor Analysis (Contd.)
Methodology
Principal Component Analysis
Summarizes information in a larger set of variables to a smaller set of factors
Common Factor Analysis
Uncovers underlying dimensions surrounding the original variables

13. Factor Analysis - Example

14. Export Data Set - Illustration
Respid Will(y1) Govt(y2) Train(x5) Size(x1) Exp(x6) Rev(x2) Years(x3) Prod(x4)

15. Description of Variables
Variable Description
Corresponding Name in Output
Scale Values
Willingness to Export (Y1)
Will
1(definitely not interested) to 5 (definitely interested)
Level of Interest in Seeking Govt Assistance (Y2)
Govt
Employee Size (X1)
Size
Greater than Zero
Firm Revenue (X2)
Rev
In millions of dollars
Years of Operation in the Domestic Market (X3)
Years
Actual number of years
Number of Products Currently Produced by the Firm (X4)
Prod
Actual number
Training of Employees (X5)
Train
0 (no formal program) or 1 (existence of a formal program)
Management Experience in International Operation (X6)
Exp
0 (no experience) or 1 (presence of experience)

16. Factors Factor Eigenvalue Criteria
A variable or construct that is not directly observable but needs to be inferred from the input variables
All included factors (prior to rotation) must explain at least as much variance as an “average variable”
Eigenvalue Criteria
Represents the amount of variance in the original variables that is associated with a factor
Sum of the square of the factor loadings of each variable on a factor represents the eigenvalue
Only factors with eigenvalues greater than 1.0 are retained

17. How Many Factors - Criteria
Scree Plot Criteria
A plot of the eigenvalues against the number of factors, in order of extraction.
The shape of the plot determines the number of factors

18. How Many Factors: Criteria (Contd.)
Percentage of Variance Criteria
The number of factors extracted is determined so that the cumulative percentage of variance extracted by the factors reaches a satisfactory level
Significance Test Criteria
Statistical significance of the separate eigenvalues is determined, and only those factors that are statistically significant are retained

19. Extraction using Principal Component Method - Unrotated
Factor Loadings
Factor Score Coefficient

20. Extraction using Principal Component Method - Factor Rotation
Not significantly different from unrotated values

21. Common Factor Analysis
The factor extraction procedure is similar to that of principal component analysis except for the input correlation matrix
Communalities or shared variance is inserted in the diagonal instead of unities in the original variable correlation matrix
The total amount of variance that can be explained by all the factors in common factor analysis is the sum of the diagonal elements in the correlation matrix
The output of common factor analysis depends on the amount of shared variance

22. Common Factor Analysis – Results (Contd.)

23. Common Factor Analysis - Results

24. Common Factor Analysis – Results (Contd.)

25. Cluster Analysis
Technique for grouping individuals or objects into unknown groups.
The typical criterion used in cluster analysis is distance between clusters or the error sum of squares.
The input is any valid measure of similarity between objects, such as:
Correlations
Distance measures (Euclidean distance)
Association coefficients
The number of clusters or the level of clustering

26. Steps in Cluster Analysis
Define the problem
Decide on the appropriate similarity measure
Decide on how to group the objects
Decide the number of clusters
Interpret, describe, and validate the clusters

27. Cluster Analysis (Contd.)
Hierarchical Clustering
Can start with all objects in one cluster and divide and subdivide them until all objects are in their own single-object cluster ( ‘top-down’ or decision approach)
Can start with each object in its own single-object cluster and systematically combine clusters until all objects are in one cluster (‘bottom-up’ or agglomerative approach)
Non-hierarchical Clustering
Permits objects to leave one cluster and join another as clusters are being formed
A cluster center is initially selected and all the objects within a pre-specified threshold distance are included in that cluster

28. Hierarchical Clustering
Single Linkage
Clustering criterion based on the shortest distance
Complete Linkage
Clustering criterion based on the longest distance

29. Hierarchical Clustering (Contd.)
Average Linkage
Clustering criterion based on the average distance
Ward's Method
Based on the loss of information resulting from grouping of the objects into clusters (minimize within cluster variation)

30. Hierarchical Clustering (Contd.)
Centroid Method
Based on the distance between the group centroids (the point whose coordinates are the means of all the observations in the cluster)

31. Hierarchical Cluster Analysis - Example

32. Hierarchical Cluster Analysis (Contd.)
A dendrogram for hierarchical clustering of bank data

33. Hierarchical Cluster Analysis (Contd.)

34. Criteria for Determining the Number of Clusters
Number of clusters is specified by the analyst for theoretical or practical reasons.
Level of clustering with respect to clustering criterion is specified.
Determine the number of clusters from the pattern of clusters generated. The distances between clusters or error variability measure at successive steps can be used to decide the number of clusters (from the plot of error sum of squares with the number of clusters).
The ratio of total within-group variance to between group variance is plotted against the number of clusters and the point at which an elbow occurs indicates the number of clusters.

35. Assumptions and Limitations of Cluster Analysis
The basic measure of similarity on which the clustering is based is a valid measure of the similarity between the objects.
There is theoretical justification for structuring the objects into clusters
Limitations
It is difficult to evaluate the quality of the clustering
It is difficult to know exactly which clusters are very similar and which objects are difficult to assign.
It is difficult to select a clustering criterion and program on any basis other than availability.