The Cycling of a Decision Threshold: A System Dynamics Model of the Taylor Russell Diagram Elise Axelrad Weaver, Ph.D. and George Richardson, Ph.D. Center for Policy Research April 20, 2001

The Cycling of Decision Thresholds In his book, Hammond (1996) presents the following ideas: Any decision threshold based on a statistically uncertain measure will inevitably yield some error and injustice in policy outcomes (a duality of error: false positives and false negatives) Oscillations in public and professional attitudes (with implicit policy thresholds) exist: Schlesinger’s (1986) proposal of “regular oscillations” in the dominance of political parties Oscillations between cautious conservatism and risky innovation in bridge design, as “the accumulation of successful experience” makes designers bold until a “collosal failure” takes everyone by surprise Cycles tend to be around 30 years long, across decision domains Hammond suggests that oscillations may be a result of the duality of error

Hammond’s exhortation: “For if such oscillations can be shown to exist, and if they can be shown to have a definite period...then we have at hand not only a means for predicting our future political climate far in advance, but an important phenomenon that strongly invites, indeed, demands, analysis and interpretation.” We propose a system dynamics model to represent and explore Hammond’s idea in a rigorous way

Another Example of a Reversal in Policy Formation Use of SAT testing in admissions 1967 University of California began using SAT scores in admissions decisions 1999 University of California faculty voted their preference to exclude SAT scores from admissions decisions 2001 Richard Atkinson, University of California President, publicly advocates removing SATs from admissions.

Example of a Decision Threshold for Policy Formation Illustration: Students are admitted to an academic program partially according to SAT score set at a given threshold The SAT score is used to predict academic success The GPA at graduation is the measure of true academic success

Taylor Russell Diagram r =.5, cutoff = 50 Positive on testNegative on test Graduates Non- Graduates Decision Threshold False + False -

Taylor Russell Diagram r =.5, cutoff = 80 Positive on testNegative on test Decision Threshold False + False - Graduates Non- Graduates

Taylor Russell Diagram r =.5, cutoff = 20 Positive on testNegative on test Decision Threshold False + False - Graduates Non- Graduates

Taylor Russell Diagram Highly Certain Test (r =.99) Positive on testNegative on test Decision Threshold False + False - Graduates

They want to reduce the number of individuals for whom the decision to accept was falsely negative: high potential for success but unacceptable SAT scores Stakeholders in the Duality of Error Constituency Concerned with Unfair Disadvantage: Constituency Concerned with Maintaining Standards: They want to reduce the number of individuals for whom the decision to accept was falsely positive: low potential for success but acceptable SAT scores

Duality of Error Due to the Decision Threshold Decision Threshold False + False -

Decision Threshold as a Stock (accumulating increases and decreases)

Decision Threshold Responding to Stakeholder Pressure Disadvantaged Constituency (DC) High Standards Constituency (HSC)

Cycling of Policy Threshold: Historic Discontent Disadvantaged Constituency (DC) High Standards Constituency (HSC)

Cycling of Policy Threshold: Key Parameters For both constituencies: Tolerated Number of False Cases Relative Weight on History Time to Respond to Pressure Threshold Change per Unit Pressure Disadvantaged Constituency (DC) High Standards Constituency (HSC)

Cycling of Policy Threshold: More Key Parameters Time Constants (Forgetting, Retention) Initial value of Threshold Lookup functions (correlation) Dissatisfaction per Error

Unexplored Parameters Lookup Function –translation of threshold choice to number of errors –involves correlation, thresholds for judgment and success –currently set at r = 0.7, variable threshold for judgment, and threshold for success fixed. Pace of Error Generation and Error Detection –Baseline model assumes that in every month, there is an assessment of number of errors due to threshold choice Threshold Associated with “Success” –Baseline model assumes a fixed threshold of “success”

Graph of Cycling of Policy Threshold SAT Score Baseline Case

Threshold Change per Unit Pressure Time to Respond to Pressure - represents policy makers’ responsiveness to constituent pressure: more responsiveness means wider amplitude and very slightly lower frequency

Initial Threshold Value - represents policy makers’ initial setting for decision threshold: no long term effect

Tolerated Number of False Cases - represents constituents’ sensitivity to errors not a key difference in the long run for either frequency or amplitude

Time Constant for Forgetting - represents time it takes for an error to dissipate from constituent memory more time to forget lowers frequency and raises amplitude

Relative Weight of History - represents constituents’ memory or weighting for accumulated past errors relative to current error more weight on history means lower frequency and higher amplitude of cycling; no weight on history means no cycling at all.

Different Weight on History for Each Constituency - represents differential weight on history by constituents - if one group puts more weight on history, it doesn’t change the center of the cycling, but it does reduce the amplitude.

Key Parameter Summary Parameters not affecting frequency: initial decision threshold policy maker sensitivity: increases amplitude,but doesn’t affect frequency much in the long run tolerated number of errors by constituents: does not have much effect on frequency or amplitude Parameters affecting frequency relative weight on history & time to forget: longer memories lower frequency and raise amplitude.

Next Steps Ground the conceptual model in data from one or more institutions (especially look for cases where false negatives could be detected) Incorporate the effects of correlations between the judgment and success measures, time to generate and detect errors, and the threshold for success Explore and operationalize the conversion variables (from errors to stakeholder pressure to changes in threshold) Consider whether there is any impact of true negatives and true positives on cycling Explore limit cycle and open loop structure of model

Summary A systems dynamics model is under development. According to Hammond (1996), any uncertain test where a threshold is used as a policy decision tool leads to unavoidable injustice to some constituency. The pressure on the decision threshold from stakeholders representing the false positives and the false negatives will oppose. These opposing pressures will cause a cycling of the decision threshold over time. A key parameter affecting frequency and amplitude of cycling is constituent weighting of past errors.