A Ratio. A Ratio. My Kingdom for a Ratio James Shanteau Kansas State University

My Problem* My problem is that I have been persecuted by a math-ematical formula. For years, this formula has followed me around, has intruded in my most private analyses, and has assaulted me from the pages of our most public journals. This formulation assumes a variety of disguises, being sometimes expressed openly, other times not, but never changing so much as to be unrecognizable. The persis-tence with which this formulation plagues me is far more than a random accident. There is a design behind it, some pattern governing its appearances. Either there is some-thing unusual about the formulation or else I am suffering from delusions of persecution. (*With apologies to George A. Miller)

My Problem* My problem is that I have been persecuted by a math-ematical formula. For years, this formula has followed me around, has intruded in my most private analyses, and has assaulted me from the pages of our most public journals. This formula assumes a variety of disguises, being sometimes expressed openly, other times not, but never changing so much as to be unrecognizable. The persis-tence with which this formula plagues me is far more than a random accident. There is a design behind it, some pattern governing its appearances. Either there is some-thing unusual about the formula or else I am suffering from delusions of persecution. (*With apologies to George A. Miller)

Examples • Speed-Accuracy Tradeoffs • Intelligence Quotient • Size-Distance Constancy • Performance = Motivation x Ability • Signal-Noise Ratio • Cost-Benefit Analysis • Input-Output Analyses • Price-Quality Relation • F-Ratio & Chi-Square

Examples • Speed-Accuracy Tradeoffs • Intelligence Quotient • Size-Distance Constancy • Performance = Motivation x Ability • Signal-Noise Ratio • Cost-Benefit Analysis • Input-Output Analyses • Price-Quality Relation • F-Ratio & Chi-Square * Bayes Theorem!!

Commonalities In these (& many more) cases, the common element is that the formula takes the form of a Ratio Score.

Commonalities In these (& many more) cases, the common element is that the formula takes the form of a Ratio Score. Note: Log of a Ratio = Difference Score, so Change Scores, Before-After Comparisons, & other Delta Scores can also can be seen as Ratio Scores.

Commonalities In these (& many more) cases, the common element is that the formula takes the form of a Ratio Score. Note: Log of a Ratio = Difference Score, so Change Scores, Before-After Comparisons, & other Delta Scores can also can be seen as Ratio Scores. Another commonality is that ratios represent tradeoffs between incompatible goals, ie, ratios are used to describe tradeoffs.

Three Examples Three personal examples illustrate advantage of ratio (tradeoff) scores over simple (absolute) scores:

Three Examples Three personal examples illustrate advantage of ratio (tradeoff) scores over simple (absolute) scores: Replacement of Absolute Thresholds by Signal Detection Theory (SDT)

Three Examples Three personal examples illustrate advantage of ratio (tradeoff) scores over simple (absolute) scores: Replacement of Absolute Thresholds by Signal Detection Theory (SDT) (2) Switch from Learning Outcomes to Change Scores = Amount Gained

Three Examples Three personal examples illustrate advantage of ratio (tradeoff) scores over simple (absolute) scores: Replacement of Absolute Thresholds by Signal Detection Theory (SDT) (2) Switch from Learning Outcomes to Change Scores = Amount Gained (3) Change from % correct scoring for evaluat-ing experts to CWS Performance Measure

Signal Detection Theory In traditional psychophysics, eg, thresholds were measured as absolute values or hits. However, this approach failed to take false alarms into account.

Signal Detection Theory In traditional psychophysics, eg, thresholds were measured as absolute values or hits. However, this approach failed to take false alarms into account. SDT incorporates both, in that d’ = f(Hits) / f(FA) = Discrimination This changes the concept of thresholds from a fixed value to a relative score.

Input-Output Analyses Traditional measures of learning look at outcome scores, eg, on standardized tests Such scores are used for promotions (or not) for teachers & changes to pedagogy But better students will out-perform weaker students, regardless of teacher or pedagogy

Input-Output Analyses Traditional measures of learning looked just at outcome scores, eg, on standardized tests Such scores are used for promotions (or not) for teachers & changes to pedagogy But better students will out-perform weaker students, regardless of teacher or pedagogy Need to look instead at Performance Gains by Input-Output Analyses Note: Nominee to Head Department of Educa-tion did not understand this difference

CWS Evaluation of Expertise typically based on % correct scores compared to “gold standards” Problems: (1) Experts work in settings where correct answers don’t exist, (2) Perfor-mance can improve beyond 100% correct, (3) Correct answers change over time.

CWS Evaluation of Expertise typically based on % correct scores compared to “gold standards” Problems: (1) Experts work in settings where correct answers don’t exist, (2) Perfor-mance can improve beyond 100% correct, (3) Correct answers change over time. CWS looks at properties of performance, rather than agreement with “gold standard” answers CWS = Discrimination/Consistency

Normalization vs Tradeoffs Ratios can result when scores are normalized, eg, test scores expressed as percentiles IQ scores normalized for age of children IQ = (Mental Age / Chronological Age) x 100 Note: Adult IQ scores reflect group norms My concern is with tradeoff ratios, ie, where decisions are made, not normalization

Implicit vs Explicit Tradeoffs Implicit tradeoffs = environmental constraints eg, Price-Quality Tradeoffs occur because it is normally impossible to get high quality at low prices

Implicit vs Explicit Tradeoffs Implicit tradeoffs = environmental constraints eg, Price-Quality Tradeoffs occur because it is normally impossible to get high quality at low prices Explicit tradeoffs = human constraints, eg, Speed-Accuracy Tradeoffs occur when decision are made about how much accuracy to give up to gain higher speed

Generalized Formulation Generalized Ratio = f (good stuff / bad stuff), or = f (good stuff / total stuff), where total stuff = good stuff + bad stuff Either way, ratios reflect tradeoffs

Generalized CWS Ratio Cochran-Weiss-Shanteau (CWS) describes performance as function of Discrimination & Consistency, where CWS = Discrimination / Inconsistency Note: reverse scoring of denominator necessary so that higher scores are better

Generalized CWS Ratio Cochran-Weiss-Shanteau (CWS) describes performance as function of Discrimination & Consistency, where CWS = Discrimination / Inconsistency Note: reverse scoring of denominator necessary so that higher scores are better Generalized CWS = S Good / S Bad This allows for other benefits & costs, eg, time & money

Conclusions Making tradeoffs is key step in almost all interesting decisions, esp. by Experts. Such tradeoffs captured by ratios.

Conclusions Making tradeoffs is key step in almost all interesting decisions, esp. by Experts. Such tradeoffs captured by ratios. When tradeoffs ignored (or overlooked), important components of problem missing. As a rule-of-thumb, fields advance when tradeoff relations recognized, eg, SDT.

Final Comments & Questions?

Ratios in Statistics Most statistical tests are based on ratios, eg, • F-Ratios (& t-tests) • Chi-Square • Bayes Likelihood Ratio

Ratios in Statistics Most statistical tests are based on ratios, eg, • F-Ratios (& t-tests) • Chi-Square • Bayes Likelihood Ratio Generally, these are • Variance ratios, where Variance Accounted For / Total Variance However, statistical ratios are not of concern for analysis of cognitive tradeoffs

Dominance (Pareto) Frontier For a set of options with negative correlations between attributes (eg, due to price-quality) the best options defined by dominance, ie, they dominant weak options.

Dominance (Pareto) Frontier For a set of options with negative correlations between attributes (eg, due to price-quality) the best options defined by dominance, ie, they dominant weak options. Thus, there are environmental limits to trade-offs that are possible, eg, between speed and fuel efficiency.

James Shanteau Kansas State University