1. Pat Langley Institute for the Study of Learning and Expertise 2164 Staunton Court, Palo Alto, California and School of Computing and Informatics Arizona State University, Tempe, AZ http://www.isle.org/~langley An Interactive Environment for Scientific Model Discovery Thanks to N. Asgharbeygi, K. Arrigo, D. Billman, S. Borrett, W. Bridewell, S. Dzeroski, J. Sanchez, and L. Todorovski for their contributions to this research, which is funded by a grant from the National Science Foundation.

2. Scientific Discovery as Problem Solving attempts to understand the nature of scientific creativity attempts to understand the nature of scientific creativity by replicating episodes from the history of science by replicating episodes from the history of science by generating entirely new scientific knowledge by generating entirely new scientific knowledge produces knowledge in formalisms used by scientists produces knowledge in formalisms used by scientists descriptive laws (qualitative and quantitative) descriptive laws (qualitative and quantitative) explanatory models (structural and process) explanatory models (structural and process) treats discovery as a form of everyday problem solving treats discovery as a form of everyday problem solving laws and models are mental problem states laws and models are mental problem states discovery occurs through heuristic search discovery occurs through heuristic search Computational scientific discovery is a research movement that: When Simon (1966) first proposed this view, it seemed radical; now it has received wide acceptance.

3. Computational Research on Scientific Discovery 1989199019791980198119821983198419851986198719881991199219931994199519961997199819992000 Bacon.1–Bacon.5 Abacus, Coper Fahrehneit, E*, Tetrad, IDS N Hume, ARC DST, GP N LaGrange SDS SSF, RF5, LaGramge Dalton, Stahl RL, Progol Gell-Mann BR-3, Mendel Pauli Stahlp, Revolver Dendral AM GlauberNGlauber IDS Q, Live IE Coast, Phineas, AbE, Kekada Mechem, CDP Astra, GP M HR BR-4 Numeric lawsQualitative lawsStructural modelsProcess models Legend Few efforts have attempted to model the details of human discovery either historically or in the laboratory. Few efforts have attempted to model the details of human discovery either historically or in the laboratory. Most work has aimed to automate discovery, rather than to produce computational aids. Most work has aimed to automate discovery, rather than to produce computational aids.

4. Observations from the Ross Sea

5. A Model of the Ross Sea Ecosystem d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus Differential equation models of this sort are regularly used to explain observations and predict future behavior.

6. The Task of Model Construction Environmental scientists are confronted with a challenging task: Given: A set of variables of interest to the scientist; Given: A set of variables of interest to the scientist; Given: Observations of how these variables change over time; Given: Observations of how these variables change over time; Find: A model that explains these variations in plausible terms and that generalizes well to future observations. Find: A model that explains these variations in plausible terms and that generalizes well to future observations. Automating such model construction is a natural task for artificial intelligence and machine learning. We can develop algorithms that search the space of differential equation models, but this space is huge, so we need constraints.

7. Another Account of the Ross Sea Ecosystem d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus As phytoplankton uptakes nitrogen, its concentration increases and nitrogen decreases. This continues until the nitrogen supply is exhausted, which leads to a phytoplankton die off. This produces detritus, which gradually remineralizes to replenish the nitrogen. Zooplankton grazes on phytoplankton, which slows the latters increase and also produces detritus.

8. Processes in the Ross Sea Ecosystem d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus Here we highlight the terms related to phytoplantkon loss, which decreases phyto concentration and increases detritus. Knowledge about candidate processes requires that some terms occur either together or not at all.

9. d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus Processes in the Ross Sea Ecosystem We can use knowledge about processes to reorganize models and constrain search through the model space. Here we highlight terms related to zooplankton grazing, which decreases phyto but increases zoo and detritus.

10. A Process Model for the Ross Sea model Ross_Sea_Ecosystem variables: phyto, zoo, nitro, detritus observables: phyto, nitro process phyto_loss equations:d[phyto,t,1] = 0.307 phyto equations:d[phyto,t,1] = 0.307 phyto d[detritus,t,1] = 0.307 phyto process zoo_loss equations:d[zoo,t,1] = 0.251 zoo equations:d[zoo,t,1] = 0.251 zoo d[detritus,t,1] = 0.251 zoo process zoo_phyto_grazing equations:d[zoo,t,1] = 0.615 0.495 zoo equations:d[zoo,t,1] = 0.615 0.495 zoo d[detritus,t,1] = 0.385 0.495 zoo d[phyto,t,1] = 0.495 zoo process nitro_uptake equations:d[phyto,t,1] = 0.411 phyto equations:d[phyto,t,1] = 0.411 phyto d[nitro,t,1] = 0.098 0.411 phyto process nitro_remineralization; equations:d[nitro,t,1] = 0.005 detritus equations:d[nitro,t,1] = 0.005 detritus d[detritus,t,1 ] = 0.005 detritus This model is equivalent to a standard differential equation model, but it makes explicit assumptions about which processes are involved. For completeness, we must also make assumptions about how to combine influences from multiple processes.

11. The Task of Inductive Process Modeling We can use these ideas to reformulate the modeling problem: Given: A set of variables of interest to the scientist; Given: A set of variables of interest to the scientist; Given: Observations of how these variables change over time; Given: Observations of how these variables change over time; Given: Background knowledge about plausible processes; Given: Background knowledge about plausible processes; Find: A process model that explains these variations and that generalizes well to future observations. Find: A process model that explains these variations and that generalizes well to future observations. We can use background knowledge about candidate processes to make search much more tractable. Moreover, the resulting model will be consistent with this domain knowledge, making it more comprehensible.

12. Generic Processes as Background Knowledge the variables involved in a process and their types; the variables involved in a process and their types; the parameters appearing in a process and their ranges; the parameters appearing in a process and their ranges; the forms of conditions on the process; and the forms of conditions on the process; and the forms of associated equations and their parameters. the forms of associated equations and their parameters. We cast background knowledge as generic processes that specify: Generic processes are building blocks from which one can compose a specific process model.

13. Generic Processes for Aquatic Ecosystems generic process exponential_lossgeneric process remineralization variables: S{species}, D{detritus} variables: N{nutrient}, D{detritus} variables: S{species}, D{detritus} variables: N{nutrient}, D{detritus} parameters: [0, 1] parameters: [0, 1] parameters: [0, 1] parameters: [0, 1] equations:d[S,t,1] = 1 S equations:d[N, t,1] = D equations:d[S,t,1] = 1 S equations:d[N, t,1] = D d[D,t,1] = Sd[D, t,1] = 1 D generic process grazinggeneric process constant_inflow variables: S1{species}, S2{species}, D{detritus} variables: N{nutrient} variables: S1{species}, S2{species}, D{detritus} variables: N{nutrient} parameters: [0, 1], [0, 1] parameters: [0, 1] parameters: [0, 1], [0, 1] parameters: [0, 1] equations:d[S1,t,1] = S1 equations:d[N,t,1] = equations:d[S1,t,1] = S1 equations:d[N,t,1] = d[D,t,1] = (1 ) S1 d[S2,t,1] = 1 S1 generic process nutrient_uptake variables: S{species}, N{nutrient} variables: S{species}, N{nutrient} parameters: [0, ], [0, 1], [0, 1] parameters: [0, ], [0, 1], [0, 1] conditions:N > conditions:N > equations:d[S,t,1] = S equations:d[S,t,1] = S d[N,t,1] = 1 S Our current library contains about 20 generic processes, including ones with alternative functional forms for loss and grazing processes.

14. process exponential_growth variables: P {population} variables: P {population} equations: d[P,t] = [0, 1, ] P equations: d[P,t] = [0, 1, ] P process logistic_growth variables: P {population} variables: P {population} equations: d[P,t] = [0, 1, ] P (1 P / [0, 1, ]) equations: d[P,t] = [0, 1, ] P (1 P / [0, 1, ]) process constant_inflow variables: I {inorganic_nutrient} variables: I {inorganic_nutrient} equations: d[I,t] = [0, 1, ] equations: d[I,t] = [0, 1, ] process consumption variables: P1 {population}, P2 {population}, nutrient_P2 variables: P1 {population}, P2 {population}, nutrient_P2 equations: d[P1,t] = [0, 1, ] P1 nutrient_P2, equations: d[P1,t] = [0, 1, ] P1 nutrient_P2, d[P2,t] = [0, 1, ] P1 nutrient_P2 d[P2,t] = [0, 1, ] P1 nutrient_P2 process no_saturation variables: P {number}, nutrient_P {number} variables: P {number}, nutrient_P {number} equations: nutrient_P = P equations: nutrient_P = P process saturation variables: P {number}, nutrient_P {number} variables: P {number}, nutrient_P {number} equations: nutrient_P = P / (P + [0, 1, ]) equations: nutrient_P = P / (P + [0, 1, ]) Constructing Process Models model AquaticEcosystem variables: nitro, phyto, zoo, nutrient_nitro, nutrient_phyto observables: nitro, phyto, zoo process phyto_exponential_growth equations: d[phyto,t] = 0.1 phyto equations: d[phyto,t] = 0.1 phyto process zoo_logistic_growth equations: d[zoo,t] = 0.1 zoo / (1 zoo / 1.5) equations: d[zoo,t] = 0.1 zoo / (1 zoo / 1.5) process phyto_nitro_consumption equations: d[nitro,t] = 1 phyto nutrient_nitro, equations: d[nitro,t] = 1 phyto nutrient_nitro, d[phyto,t] = 1 phyto nutrient_nitro d[phyto,t] = 1 phyto nutrient_nitro process phyto_nitro_no_saturation equations: nutrient_nitro = nitro equations: nutrient_nitro = nitro process zoo_phyto_consumption equations: d[phyto,t] = 1 zoo nutrient_phyto, equations: d[phyto,t] = 1 zoo nutrient_phyto, d[zoo,t] = 1 zoo nutrient_phyto d[zoo,t] = 1 zoo nutrient_phyto process zoo_phyto_saturation equations: nutrient_phyto = phyto / (phyto + 0.5) equations: nutrient_phyto = phyto / (phyto + 0.5) HeuristicSearch observations generic processes process model phyto, nitro, zoo, nutrient_nitro, nutrient_phyto variables

15. A Method for Process Model Construction 1. Find all ways to instantiate known generic processes with specific variables, subject to type constraints; 2. Combine instantiated processes into candidate generic models subject to additional constraints (e.g., number of processes); 3. For each generic model, carry out search through parameter space to find good coefficients; 4. Return the parameterized model with the best overall score. Our initial system, IPM, constructs process models from generic components in four stages: Our typical evaluation metric is squared error, but we have also explored other measures of explanatory adequacy.

16. Results on Observations from Ross Sea We provided IPM with 188 samples of phytoplnkton, nitrate, and ice measures taken from the Ross Sea. From 2035 distinct model structures, it found accurate models that limited phyto growth by the nitrate and the light available. Some high-ranking models incorporated zooplankton, whereas others did not.

17. Results with Inductive Process Modeling population dynamics battery behavior hydrology biochemical kinetics

18. Extensions to Inductive Process Modeling heuristic beam search through the space of process models; heuristic beam search through the space of process models; hierarchical generic processes that further constrain search; hierarchical generic processes that further constrain search; an ensemble-like method that mitigates overfitting effects; an ensemble-like method that mitigates overfitting effects; an EM-like method that deals with missing observations. an EM-like method that deals with missing observations. In recent work, we have extended our system to incorporate: This approach has great potential to speed the construction of scientifc models – provided that domain users adopt it.

19. specify a quantitative process model of the target system; specify a quantitative process model of the target system; display and edit the models structure and details graphically; display and edit the models structure and details graphically; simulate the models behavior over time and situations; simulate the models behavior over time and situations; compare the models predicted behavior to observations; compare the models predicted behavior to observations; invoke a revision module in response to detected anomalies. invoke a revision module in response to detected anomalies. Because few scientists want to be replaced, we are developing an interactive environment, P ROMETHEUS, that lets users: The environment offers computational assistance in forming and evaluating models but lets the user retain control. Interfacing with Scientists

20. Viewing a Process Model Graphically

21. Viewing a Process Model as Equations

22. Adding a Process Manually

23. Requesting Automatic Model Revision

24. Results of Automatic Model Revision

25. liked the ability to alter models at an abstract process level; liked the ability to alter models at an abstract process level; appreciated the ability to simulate models and plot variables; appreciated the ability to simulate models and plot variables; believed the environment would be a good teaching tool; believed the environment would be a good teaching tool; had some confusion about edge semantics in graphical display; had some confusion about edge semantics in graphical display; had interference from STELLAs stock and flow approach; had interference from STELLAs stock and flow approach; felt system would be more useful if it supported PDE models. felt system would be more useful if it supported PDE models. An Initial User Study We asked three oceanographers to use P ROMETHEUS to revise a model of the Ross Sea ecosystem; this study revealed that they: We plan to address the last three issues in our next version of the modeling environment.

26. Directions for Future Research provide better ways to visualize models, data, and their relation provide better ways to visualize models, data, and their relation offer users more natural ways to define the space of models offer users more natural ways to define the space of models specifying constraints on relations among entities and processes specifying constraints on relations among entities and processes characterizing subsystems that decompose complex models characterizing subsystems that decompose complex models support finer-grained control of search through model space support finer-grained control of search through model space incorporate intuitive metrics like match to trajectory shape incorporate intuitive metrics like match to trajectory shape more generally improve the usability of P ROMETHEUS more generally improve the usability of P ROMETHEUS Despite our progress to date, we need further work in order to: Taken together, these will make inductive process modeling a more robust approach to scientific model construction.

27. computational scientific discovery (e.g., Langley et al., 1983); computational scientific discovery (e.g., Langley et al., 1983); theory revision in machine learning (e.g., Towell, 1991); theory revision in machine learning (e.g., Towell, 1991); qualitative physics and simulation (e.g., Forbus, 1984); qualitative physics and simulation (e.g., Forbus, 1984); languages for scientific simulation (e.g., STELLA, MATLAB ); languages for scientific simulation (e.g., STELLA, MATLAB ); interactive tools for data analysis (e.g., Schneiderman, 2001). interactive tools for data analysis (e.g., Schneiderman, 2001). Intellectual Influences Our approach to aiding scientific model construction incorporates ideas from many traditions: Our work combines, in novel ways, insights from machine learning, AI, programming languages, and human-computer interaction.

28. Contributions of the Research a new formalism for representing scientific process models; a new formalism for representing scientific process models; a computational method for simulating these models behavior; a computational method for simulating these models behavior; an encoding for background knowledge as generic processes; an encoding for background knowledge as generic processes; algorithms for inducing process models from time-series data; algorithms for inducing process models from time-series data; an interactive environment for model construction/utilization. an interactive environment for model construction/utilization. In summary, our work on computational discovery has produced: We believe that P ROMETHEUS offers a promising approach for supporting innovative development of scientific models.

29. In Memoriam Herbert A. Simon (1916 – 2001) Herbert A. Simon (1916 – 2001) Jan M. Zytkow (1945 – 2001) Jan M. Zytkow (1945 – 2001) Five years ago, the field of computational scientific discovery lost two of its founding fathers: Both contributed to the field in many ways: posing new problems, inventing methods, training students, and organizing meetings. Moreover, both were interdisciplinary researchers who contributed to computer science, psychology, philosophy, and statistics. Herb Simon and Jan Zytkow were excellent role models who we should all aim to emulate.

30. End of Presentation

31. Relevant Historical Events 1956 – Newell, Shaw, and Simon develop the Logic Theorist, which uses heuristic search to prove theorems 1956 – Newell, Shaw, and Simon develop the Logic Theorist, which uses heuristic search to prove theorems 1966 – Simon proposes computational modeling of scientific discovery as problem solving through heuristic search 1966 – Simon proposes computational modeling of scientific discovery as problem solving through heuristic search Late 1970s – Lenat and Langley use this framework to reproduce episodes from the history of mathematics and physics Late 1970s – Lenat and Langley use this framework to reproduce episodes from the history of mathematics and physics 1987 – Langley, Simon, Bradshaw, and Zytkow report computational accounts for a wide range of historical discoveries 1987 – Langley, Simon, Bradshaw, and Zytkow report computational accounts for a wide range of historical discoveries 1990 – Valdes-Perez extends this approach to explanatory process models in physical chemistry 1990 – Valdes-Perez extends this approach to explanatory process models in physical chemistry 1994 onward – Researchers adapt these methods to discover knowledge in many scientific domains, leading to refereed publications 1994 onward – Researchers adapt these methods to discover knowledge in many scientific domains, leading to refereed publications 1995 onward – The data mining movement redirects attention toward business applications and purely predictive models 1995 onward – The data mining movement redirects attention toward business applications and purely predictive models 2006 – Challenges remain in constructing understandable scientific models 2006 – Challenges remain in constructing understandable scientific models