1 Some Metrological Aspects of Ordinal Quality Data Treatment *Emil Bashkansky Tamar Gadrich ORT Braude College of Engineering, Israel  ENBIS-11 Coimbra, Portugal,  September 2011, 11:50 – 13:20

2 Presentation Outline-stage I Why revision of MC for ordinal measurements is needed? Error Binary caseGeneral case Uncertainty General caseBinary case

3 Presentation Outline-stage II Main metrological characteristics AccuracyPrecision RepeatabilityReproducibility

4 Presentation Outline-stage III Repeated measurements Binary caseGeneral case

5 Examples of ordinal scale usage DAILY LIFE MEDICINE QUALITY MANAGEMENT Engineering Sports results: a win, tie, loss Voting results: pro, against, abstain Academic ranks … Rankin score (RS) - level of disability following a stroke Side effect severity … Quality level estimation and sorting Customer satisfaction surveys Ratings of wine colour, aroma and taste FMECA … The Mohs scale of mineral hardness Dry-chemistry dipsticks (e.g., urine test) The Beaufort wind force scale....

6 Why revision of MC for ordinal measurements is needed? * ISO/IEC Guide 99: International vocabulary of metrology — “Basic and general concepts and associated terms (VIM)”

7 Classic continual: The probability density function pdf (Y/X) of receiving result Y, given the true value of the measurand X, in it's simplest form: pdf (Y/X) = Normal (X+bias, ) Ordinal: The conditional probabilities that an object will be classified as level j, given that its actual/true level is i. Error description

8 Error -free ordinal measurement

9 Error –binary case

10 Some examples

11 Uncertainty-general case The likelihood that a measured level j is received, whereas the true level is i

12 Uncertainty matrix- binary case

13 Inaccuracy- Error matrix :

14 Effectiveness vs. Accuracy Effectiveness of the measurement system Bashkansky E, Dror S, Ravid R, Grabov P (2007) Effectiveness of a Product Quality Classifier. Quality Engineering 19(3):

15 Precision (the closeness of agreement between independent test results obtained under stipulated conditions) Repeatability (same conditions) Reproducibility (different conditions)

16 Repeatability Blair & Lacy (2000)

17 Repeatability - the expected cumulative frequency of data/items classified up to the k-th category, given that its actual/true level is i

18 - the expected cumulative frequency of items belonging up to the k-th category after measurement ORDANOVA: DECOMPOSITION OF TOTAL DISPERSION AFTER MEASUREMENT/CLASSIFICATION

19 Reference standard (known/unknown) Measurement system A Measurement system B

20 Some definitions - conditional joint probability of sorting the measured object to the a-th level by the first MS (called A), and the b-th level by the second MS (called B), given the actual/true category i

21 A & B MSs classification matrices

22 Joint probability matrix p i - the probability that an object being measured relates to category i, ( ) - the joint probability of sorting the item as a by the first measurement system (A) and b by the second measurement system (B).

23 MODIFIED KAPPA MEASURE OF AGREEMENT m When a half of all items are correctly classified:

24 1. Reproducibility - reference standard is known

25 2. Reproducibility - reference standard is unknown

26 Binary case example

27 QUALITY CATEGORY SOLUBLE SOLIDS CONTENT (SSC) TITRATABLE ACIDITY (TA) PH TOTAL SUGARS MASS FRACTIONS SKIN COLOR ("A" VALUE) FLESH FIRMNESS MASS (%) (G/KG) (N)(G) HIGH (TYPE 1) >105 MEDIUM (TYPE 2) LOW (TYPE 3) <12>1.1<3.6<50>25>55<85 Typical relation between quality level and commonly used chemical/physical features for yellow-flesh nectarines

28 weighted total kappa equals Ternary scale example (fruit quality classification)

29 Repeated measurements-binary case (n = n 1 +n 2 )

30 Repeated measurements-binary case αβ CUT POINT

31 Binary case example

32 LIKELIHOODS (given the true value one or two) vs. n 1 /n LIKELIHOODS (given the true value one or two) vs. n 1 /n

33 Repeated measurements-general case Let's consider arbitrary ordinal scale with m categories and suppose, that n repeated measurements of the same object were performed resulting in vector: (n=n 1 +n 2 +…+n m )

34 Repeated measurements-general case The maximum likelihood estimation must be made in favor of such, most plausible i, that maximizes the scalar product:

35 General case example-single measurement

36 General case example: after 10 repititions

37 Binary case free access calculator

38SUMMARY 1.On an ordinal measurement scale the essential for evaluating the error, repeatability and uncertainty of the measurement result base knowledge must be the classification/measurement matrix. Given this matrix, authors introduced a way to calculate the classification/measurement system’s accuracy, precision (repeatability & reproducibility) and uncertainty matrix. 2.In order to estimate comparability and equivalence between measurement results received on an ordinal scale basis, the modified kappa measure is suggested. Three of the most suitable usages of the measure were thoroughly analyzed. The advantage of the proposed measure vs. the traditional one lies in the fact that the former follows the superposition principle: the total measure equals the weighted sum of partial measures for every ordinal category. 3.As it is well known, repeated measurements may improve the quality of the measurement result. When decisions are ML based, one can find how many repetitions are necessary in order to achieve the desired accuracy level using the algorithm suggested by the authors,.

39 1.Bashkansky E., Dror S., Ravid R., Grabov P. (2007), “Effectiveness of a Product Quality Classifier”, Quality Engineering, vol. 19, issue 3, pp Bashkansky E., Gadrich T., (2010) “Some Metrological Aspects of Ordinal Measurements”, Accreditation and Quality Assurance, vol. 15, pp Bashkansky E., Gadrich T., Knani D., (2011) “Some metrological aspects of the comparison between two ordinal measuring systems”, Accreditation and Quality Assurance, vol. 16, pp

40

41