Rick Hoyle Duke Dept. of Psychology & Neuroscience rhoyle@duke.edu Is Observation Oriented Modeling a Way Forward for the Behavioral Sciences? Rick Hoyle Duke Dept. of Psychology & Neuroscience rhoyle@duke.edu

Crisis of confidence in psychological science 2011 – Stapel suspended for fabricating data used in research publications leading to 58 retractions 2011 – Bem’s paper reported findings from 9 experiments seeming to show pre-cognition 2012 – John et al. publish findings from survey or researchers showing that more than half routinely engage in questionable research practices 2011 – Bakker & Wicherts reported widespread statistical reporting errors, some of which, if corrected, would change the inference

Are the prevailing research methods capable of yielding an authentic science of psychology? [I]s significance testing the appropriate tool for evaluating data? Is the randomized controlled experiment sufficient for determining causality? Are parametric statistics appropriate for the attributes studied by psychologists? Grice (2011, p. 2)

Problems to be solved concerns about pervasive Type 1 errors uncertainty about reproducibility of effects concerns about validity of measurement dearth of translatable research outcomes that could inform policy and practice It’s time for something new!

OOM is an appealing alternative to the status quo more than just an attempt to fix what is broken shifts the focus from population focused group-level patterns and comparisons and variable based data modeling to observation-focused methods What might a specific person think or feel or do under certain circumstances? “. . . observation oriented modeling shifts the focus of analysis away from computed aggregates such as means and variances onto the observations themselves.” (Grice, 2011, p. 45)

How do we get there from here? things to stop doing NHST treating measures as if they were continuous (or even interval scaled) testing hypotheses using summary statistics inferring from sample to population

How do we get there from here? things to start doing using techniques that do not make untenable assumptions about samples and measures addressing generalizability and reproducibility concerns through replication and theory-development rather than sampling and p-values aligning hypotheses and methods to focus directly on what individuals (rather than populations) do designing studies that involve intensive repeated assessments

The worst-case scenario is clearly dire: It is plausible that inattention to nonergodicity and a lack of group-to-individual generalizability threaten the veracity of countless studies, conclusions, and best- practice recommendations. . . . even in the best-case scenario, we should not think of a correlation in group data as an estimate that generalizes to any given individual in the population. Stated bluntly, this implies that the temptation to use aggregate estimates to draw inferences at the basic unit of social and psychological organization—the person—is far less accurate or valid than it may appear in the literature. Fisher et al. (2018)

Practical matters How receptive are journals likely to be to the OOM approach? How would findings from observation-oriented research be synthesized? Is there a place for “big data” in the behavioral sciences given the concerns that OOM addresses? Is it time to abandon classical statistics in behavioral science research?