1 Presentation Outline Introduction Objective Sorting Probability and Loss Matrices The Proposed Model Analysis Of Some Loss Functions Case Study Redundancy based methods Illustrative example Summary

2 Introduction A. Accuracy and Precision B. Types of data C. Binary Situation

3 A. Accuracy and Precision. Accuracy The closeness of agreement between the result of measurement and the true (reference) value of the product being sorted. Precision Estimate of both the variation in repeated measurements obtained under the same conditions (Repeatability) and the variation of repeated measurements obtained under different conditions (Reproducibility).

4 B. Two types of data characterizing products or processes Variables (results of measurement, Interval or Ratio Scales) Attributes (results of testing, Nominal or Ordinal Scales ).

5 Four types of data The four levels were proposed by Stanley Smith Stevens in his 1946 article.Different mathematical operations on variables are possible, depending on the level at which a variable is measured.

6 Categorical Variables 1. Nominal scale: gender, race, religious affiliation, the number of a bus. Possible operations:

7 Categorical Variables(cont.) 2. Ordinal scale : results of internet page rank, alphabetic order, Mohs hardness scale (10 levels from talc to diamond) customer satisfaction grade, quality sort, customer importance (QFD) vendor’s priority, severity of failure or RPN (FMECA), the power of linkage (QFD) Possible operations:

8 Numerical data. 3. Interval scale: temperature in Celsius or Fahrenheit scale, object coordinate, electric potential. Possible operations:

9 Numerical data (cont.) 4. Ratio scale: most physical quantities, such as mass, or energy, temperature, when it is measured in kelvins, amount of children in family, age. Possible operations:

10 C. The sorting probability matrix for the binary situation Accuracy is characterized by: 1. Type I Errors (non-defective is reported as defective) – alfa risk 2. Type II Errors (defective is reported as non-defective ) – beta risk

11 The sorting probability matrix for the binary situation (cont.)

12 Objective Developing a new statistical procedure for evaluating the accuracy and effectiveness of measurement systems applicable to Attribute Data based on the Taguchi approach.

13 Sorting Probability Matrix The sorting matrix is an 'm by m' matrix. Its components P i,j are the conditional probabilities that an item will be classified as quality level j, given its quality level is i. A stochastic matrix:

14 Four Interesting Sorting Matrices (a) The most exact sorting: (b) The uniform sorting: (designated as MDS: most disordered sorting): (c) The “worst case” sorting. For example, if m = 4:

15 Four Interesting Sorting Matrices (cont.) (d) Absence of any sorting.For example, if m = 2:

16 Indicator of the classification system inexactness

17 Loss Matrix Definition Let L ij - be the loss incurred by classifying sort i as sort j.

18 The Proposed Model Expected Loss Definition: Effectiveness Measure:

19 Analysis Of Some Loss Functions Equal loss: Quadratic loss: Entropy loss: Linear loss:

20 Equal loss If there is no preliminary information about p i

21 Equal loss (Cont.) If any preliminary information about p i is available :

22 Quadratic Loss For ordinal data, the total accuracy of the rating could be defined as the expected value of L ij.

23 Quadratic Loss (Cont.) If there is no preliminary information about p i : If there is any preliminary information about p i :

24 Entropy loss If level i is systematically related to level j (P ij =1), there is no “entropy loss”. The above loss function leads us directly to Eff = Theil’s uncertainty index.

25 Linear Loss Could be useful for bias evaluation: This measure characterizes the dominant tendency [over grading, if Bias > 0 or under grading, if Bias < 0 ]

26 Case Study (nectarines sorting) Type , Type , Type Classification TotalType3Type2Type Type1 Act ual Type Type Total

27 The Classification Matrix

28 The Loss matrix

29 Effectiveness Evaluations According to proposed approach: Eff = 80% Compare the above to the measures of effectiveness for different loss functions : Equal loss: 74%. Taguchi loss: 87%. Disorder entropy loss: 62% Linear penalty function:79% Traditional kappa measure : 79%. It can be seen that the effectiveness estimate strongly depends on the loss metric model.

30 REPEATED SORTING

31 Independency vs. Correlation

32 Case A: Two Independent Repeated Ratings Real improvement may be obtained if, in the case of disagreement, the final decision is made in favor of the inferior sort (one rater can see a defect, which the other has not detected). Usually, the loss that results from overrating is greater than the loss due to underrating, thus such a redistribution of probabilities seems to be legitimate. Nevertheless, to verify improvement in sorting effectiveness, we need a new expected loss calculation.

33 Case B: Three Repeated Ratings We add a third rater only if the first two raters do not agree. For most industrial applications this means that a product is passed through a scrupulous laboratory inspection or, for the purpose of our analysis, through an MRB board. The decision could be considered as an etalon measurement. The probability of correct decisions increases, and the probability of wrong decisions, decreases.

34 Conclusions To decide whether a double or triple rating procedure is expedient the total expenditures have to be compared

35 Case C: A Hierarchical Classification System The classification procedure is built on more than one level. To characterize such a hierarchical classification system G can be utilized as a “pure” indicator of the classification system’s inexactness. Usually, the cost of classification (COC) has an inverse relationship to the amount of G. In contrast to the COC, the expected loss usually decreases, as the exactness of the judgment improves.

36 Optimization of HCS If one decides to pass the hierarchical classification subsystem from the lower level (1) up to the K level, the total expenditure can be optimized by looking for the best level minimizing it.

37 A CASE STUDY AND ILLUSTRATIVE EXAMPLE The proportions of the sort types were: Type , Type and Type The same loss matrix was considered. From an R&R study, executed in relation to two independent raters’ results, the joint probability matrices were estimated.

38 Summary table One rater Two independent raters ( case A ) Three raters (case B) Sorting matrix 0.89, 0.08, , 0.85, , 0.14, , 0.10, , 0.80, , 0.06, , 0.04, , 0.96, , 0.92, 0.07 Exp. lossEL = 0.534EL' = 0.309EL'' = Inexact. Ind.G = G' = G'' =

39 Summary The proposed procedure for evaluation of product quality classifiers takes into account some a priori knowledge about the incoming product, errors of sorting and losses due to under/over graduation.

40 Summary (Cont.1) The appropriate choice of the loss function (matrix) provides the opportunity to fit quality sorting process model to the real situation. The effectiveness of quality classifying can be improved by different redundancy based methods. However, the advantages of redundancy based methods are not unequivocal, as is the case in the usual measurement processes, and corresponding calculations according to the technique being used are required.

41 Summary (Cont.2) The conclusion concerning the selection of the preferred case depends on the losses due to misclassification, as well as on the incoming quality sort distribution. Possible applications of the proposed approach are not limited only to quality sorting. The approach can be extended to other QA processes concerned with classification

42 Thank You