The Science/Policy Interface in Logic‑based Evaluation of Forest Ecosystem Sustainability Keith M. Reynolds, USDA Forest Service K. Norman Johnson, Oregon State University Sean N. Gordon, Oregon State University

Acknowledgments USDA Forest Service Washington Office National Forest System, Ecosystem Management Pacific Northwest Research Station Human and Natural Resource Interactions RD&A Program

Objectives Illustrate the value of a logic-based approach in designing a formal specification to evaluate the Montreal criteria and indicators. Identify the roles of science and policy in this effort. Highlight lessons learned from this process. Suggest some general recommendations.

Overview Introduction Knowledge bases and logic modeling Analysis Model design issues Lessons learned Recommendations

Introduction: criterion A standard that a thing is judged by (Prabhu et al. 2001). Criteria are the intermediate points to which the information provided by the indicators can be integrated and where an interpretable assessment crystallizes. Principles [e.g., sustainability] form the final point of integration. A criterion should be treated as a reflection of knowledge. It can be viewed as a large‑scale selective combination … of related pieces of information.

Introduction: indicators An indicator is any variable or component of the forest ecosystem … used to infer attributes of the sustainability of the resource and its utilization. Indicators should convey a ‘single meaningful message.’ This ‘single message’ is termed information. It represents an aggregate of one or more data elements with certain established relationships (Prabhu et al. 2001).

Introduction: measurement endpoint Some indicators are simple. Their definition suggests an obvious one‑to‑one correspondence between an indicator and a metric for that indicator. Definitions of some indicators are more complex. They represent a synthesis of two or more data elements, which we refer to as measurement endpoints.

Introduction: scales of application Purpose varies with scale (Castañeda 2001) National and regional Policy instruments to evaluate laws, policy, regulations E.g., Montreal Process, NWFP, ICBEMP Management unit Evaluation and adjustment of management practices E.g., CIFOR, USDA FS IMI (LUCID)

Knowledge bases A form of meta database Advantages A formal logical representation of how to evaluate information Networks of interrelated topics Mental map Advantages Interactive, graphic design (modularity) Numerous & diverse topics can be analyzed within a single integrated analysis

Knowledge bases: forms of uncertainty Probabilistic uncertainty Uncertainty of events Linguistic uncertainty Uncertainty about the definition of events Vagueness or imprecision A proposition is the smallest unit of thought to which one can assign a measure of truth (strength of evidence) SE: a measure that quantifies the degree of support for a proposition provided by its premises

Knowledge bases: networks of topics Concern 1 Concern 2 Ecostate A Ecostate B Etc. Ecostate C Ecostate D Data link Data link Data link Data link Data link Data 10 6

Knowledge bases: topics Each topic typically evaluates a proposition Attributes of topics Name (e.g., watershed condition) Proposition (e.g., WS condition is suitable) Strength of evidence for the proposition NetWeaver scale is [-1, 1] A formal logic specification (e.g., a proof) Documentation Explanation, source, citations, assumptions

Knowledge bases: evaluation Concern 1 Ecostate A Get data requirements Evaluate data Ecostate B Ecostate C Data link Data link Data link 11 7

Knowledge bases: strength of evidence

Analysis: Montreal C&I The Montreal specifications provide relatively clear definitions of biophysical, socioeconomic, and framework attributes requiring evaluation (WGCICSMTBF 1995) ... But, design of evaluation procedures that allow interpretation of the Montreal C&I is one of the major technical issues that remain to be resolved (Raison et al. 2001).

Analysis: conceptual framework Specified conditions or outcomes to be sustained (the indicators). A measure for each condition or outcome. Calculation of the level of the indicator over some time period using the selected measure. A frame of reference for gauging sustainability. Methods for evaluating sustainability (sustainability check). A monitoring program. A formalism that supports requirements 1 to 6.

Analysis: logic models as design frameworks Logic models (knowledge bases) provide a formal specification for organizing and interpreting information. NetWeaver kb developer system Problem represented in terms of propositions about topics of interest and their interdependencies. Topics translated into propositions. Lexical uncertainty.

Analysis: logic models as design frameworks (continued) Need for transparency (Prabhu et al. 2001) Models embody important policy decisions. Models depend on value judgments and critical assumptions that need clear documentation. Model development Graphic representation is an effective basis for organizing discussion and for evolution of design. Communication Between scientists and policy makers. With interested publics.

Design issues: model organization Basic organization of topics. For example, evaluation of criteria in the current prototype.

Design issues: model organization An alternative organization with very different emphasis on criteria.

Design issues: synthesis AND operator: arguments evaluated as limiting factors. SUM operator: arguments contribute incrementally to evaluation and can compensate.

Design issues: synthesis Another example, including the OR operator.

Design issues: weighting Intrinsic weights Each topic in a NetWeaver logic model has an intrinsic weight attribute. E.g., set weight attribute on any topic to adjust its contribution of evidence to a proposition. Bad idea: part of specification, but not obvious. Explicit weights Better, but add another layer of subjectivity. Some valid purposes, however.

Design issues: reference conditions Each elementary network evaluates a measurement endpoint against reference conditions. Lack of reference conditions is a basic problem for most measurement endpoints.

Design issues: reference conditions Implementation of a elementary network to evaluate measurement endpoint against reference conditions.

Design issues: qualitative measures Outcomes evaluated on an ordinal scale.

Design issues: reliability of data Reliability of data for evaluation of Montreal C&I. Stochastic, rather than lexical, uncertainty Formal representation of stochastic uncertainty is problematic in the context of a logic model. Not addressed in the current Montreal C&I prototype. Possible solution Adjusting topic weights with a normalized metric such as standard error of the mean. Problems: availability, unknown error correlations

Design issues: precision of knowledge Sequential OR (SOR) to specify multiple alternative pathways in order of preference.

Lessons learned Lexical uncertainty is an important issue in evaluation of Montreal criteria and indicators. Many aspects of evaluating sustainability cannot be answered by science alone. Acquiring data on sustainability is necessary but not sufficient for setting policy. Evaluating sustainability is not the same as defining desired future conditions. Evaluating the state of sustainability and deciding how to respond are separate but interdependent decision processes.

Recommendations Assess the policy role in sustainability evaluation, and undertake a policy review of model organization and strategies for integrating sustainability information. The clearest, and most critical, role of science is in development of reference conditions. A major effort is needed to identify measurement endpoints for indicators of the institutional framework (criterion 7).

Followup Scientist peer review (Mar 14/21, 2002) John Gordon (forester, Yale) Jerry Franklin (ecologist, U. of Washington) Hal Salwasser (Dean, CoF, Oregon State University) Denise Lach (sociologist, Oregon State University) Gordie Reeves (fisheries biologist, PNW) Richard Haynes (economist, PNW)

Followup Management peer review (Mar 14/21, 2002) Darrel Kenops (supervisor, Willamette NF) Gloria Brown (supervisor, Siuslaw NF) Ted Lorensen (manager, Oregon DoF) Jon Martin (REO monitoring, Region 6) Dick Phillips (economist, Region 6) Sarah Crim (analyst, Region 6)

Followup Demo with criterion 6 data from 2003 report Richard Haynes (PNW) Ken Skog (FPL) Sue Alexander (PNW) Others? Demo with LUCID link? Possibly July/Augest 2002

Authors Keith M. Reynolds K. Norman Johnson Sean N. Gordon USDA Forest Service, Pacific Northwest Research Stn. kreynolds@fs.fed.us K. Norman Johnson Oregon State University, College of Forestry norm.johnson@orst.edu Sean N. Gordon sean.gordon@orst.edu