Randy Keyworth Jack States Ronnie Detrich The Wing Institute Professional Judgment: Inevitability, and Manageability Fallibility,

What is professional judgment? Also known as………….. informed clinical opinion professional wisdomclinical reasoning clinical expertisedecision making clinical decision makingproblem solving clinical judgmentdata interpretation

What is professional judgment? Critical interface between input and output when making clinical decisions.  Input  information  data  research  Output  strategies  interventions  treatment

What is professional judgment? It is also…………  a complex set of behaviors  governed by individual learning histories  subject to complex contingencies

Why is professional judgment part of the hierarchy of evidence? Medicine Evidence-based medicine is the integration of best research evidence with clinical expertise and patient values. David L. Sacket Centre for Evidence Based Medicine Psychology Evidence-based practice in psychology (EBPP) is the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences. American Psychological Association Education The integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction. Grover J. Whitehurst, Assistant Secretary, United States Department of Education social validity

Why is professional judgment part of the hierarchy of evidence? social validity Professional judgment is ingrained in virtually all professional standards and is a universally accepted component of professional skills. Professional judgment is the age old “trusted method”, generally accepted, and expected by consumers. Professional judgment has tremendous influence in shaping behavior.

Why is professional judgment part of the hierarchy of evidence? Everyone relies upon professional judgment when making clinical decisions…  when scientific research evidence is absent, incomplete or conflicting in conclusions  when scientific research evidence is present (review and interpretation… single subject research)  when responding to the complexity and uniqueness of behavior inevitability

Why is professional judgment part of the hierarchy of evidence? inevitability  Professional judgment is used in making decisions on critical components of behavioral interventions. goals / priorities assessment strategies interventions data interpretation remediation  Professional judgment is used in countless moment-to- moment decisions.

Why is professional judgment part of the hierarchy of evidence?  Experience  with types of interventions, problems, strategies  with specific environments, individuals  Flexibility  when the data is limited or absent  when dealing with unique situations  Inexpensive  compared to costs of research  Immediate  often can’t wait for research to be conducted functionality

The fallibility of professional judgment: Professional Arrogance: ignoring the fact that mistakes are inevitable Art vs. Science: under and over- reliance on science Lack of Feedback: absence of accurate feedback a false sense of accuracy

The fallibility of professional judgment: a false sense of accuracy “art vs. science”  decisions made on the basis of intuition  empirical data not only waste of time, diminishes quality of service…it limits creativity and spontaneity  professionals “trust” their professional judgment

The fallibility of professional judgment: a false sense of accuracy 77% 22% Your intuition (gut feeling) about what will be effective 39% 92% Your demonstrated track record of success based on data you have gathered systematically and regularly 37%92% Results of controlled experimental Studies CRITERIA FOR TREATMENT CHOICES CLIENT PHYSICIAN Gambrill and Gibbs, 2002

The fallibility of professional judgment: a false sense of accuracy “science vs. art”  failure to recognize the limitations of existing scientific research  failure to recognize the role and fallibility of professional judgment

The fallibility of professional judgment: a false sense of accuracy Lack of Feedback  lack of systematic feedback from peers or colleagues  lack of organizational systems for providing feedback on outcomes related to decisions  difficulty with “complex systems” to ascertain cause and effect

The fallibility of professional judgment: Decision makers are subject to contingencies that shape behavior  lack of time, resources to stay up on literature and research individual cases  external contingencies (culture, organization, systems, consumers, professionals, regulations)  lack of standards for professional judgment conflicting contingencies

The fallibility of professional judgment: Biases that can interfere with judging progress and causation  Being swayed by hindsight  Being overconfident  Engaging in wishful thinking  Having an illusion of control Critical Thinking in Clinical Practice : Improving the Quality of Judgements and Decisions, Eileen Gambrill bias

The fallibility of professional judgment: bias Biases that can interfere with judging progress and causation  Overlooking the role of chance (coincidences)  Overlooking confounding causes, such as regression effects  Attributing our success to our own attributes and failure to other factors  Seeking only data that support preferred views Critical Thinking in Clinical Practice : Improving the Quality of Judgements and Decisions, Eileen Gambrill

The fallibility of professional judgment: bias Biases that can interfere with judging progress and causation  Relying on observed rather than relative frequency  Overlooking the interaction between predictions and their consequences  Mistaking correlation for causation  Relying on misleading criteria such as testimonials Critical Thinking in Clinical Practice : Improving the Quality of Judgements and Decisions, Eileen Gambrill

The fallibility of professional judgment: errors in reasoning Common errors in reasoning that can effect perceptions and decisions. Circular Reasoning Non-Sequitur Post Hoc Red Herring Equivocation False Dichotomy Lying Authority Shifting the Burden of Proof Self-Referential Fallacy Ad Hominem Sidestepping/Avoiding the Question Suppressed Evidence (Stacking the Deck) Statistics Jumping to Conclusions Traditional Wisdom Analogy Humor Extrapolation Circumstantial Evidence Straw Man Guilt by Association Best-in-Field Fallacy

The fallibility of professional judgment: propaganda Systematic manipulation of information ad hominum ad populum bandwagon big lie card stacking doublespeak either / or fear glittering generalities media manipulation opinion as fact plain folks repetition scapegoating testimonial transfer

The management of professional judgment: raising your batting average  learn about sources of error  develop strategies for minimizing mistakes  track outcomes from your decisions  modify decisions according to data  create environmental contingencies for feedback,continuous learning, and reviewing research  maintain a healthy sense of skepticism …… and humility

Evidence-based special education: Why now? Tracking the term “autism cure” in the Google search engine… August ’04: 308,000 results February ’05: 528,000 results August ’05: 1,830,000 results February ’06: 5,290,000 results April ’06: 15,200,000 results

Evidence-based special education: Why now?

GAP Analysis: Adequate Research Current Conditions: There is little agreement on what constitutes research EXPERIMENTAL Randomized Trials (Group Design) Single Subject (Individual Design) QUASI-EXPERIMENTAL Non-Random Trial (Group Design) DESCRIPTIVE Correlational Research Simple Descriptive Comparative Descriptive Case Study QUALITATIVE Case Studies Interviews Ethnography Participant Observation Document / Artifact Analysis OPINION Professional Opinion Testimonials Anecdotal

Why is professional judgment part of the hierarchy of evidence? inevitability Professional judgment is critical in making sound decisions… when addressing client-centered outcomes (patient values) “by patient values we mean the unique preferences, concerns and expectations each patient brings to a clinical encounter and which must be integrated into clinical decisions if they are to serve the patient” David Sacket (not always “ideal solutions”…context of “consumer” resources, biases, choices, preferences)

The fallibility of professional judgment: bias Questionable Criteria  authority  status is correlated with accuracy  popularity and numbers  acceptance of claims simply because many people accept them  tradition  what has been done in the past  newness  being swayed by the “latest intervention”  manner of presentation  swayed by style, persuasive style