NZSSES/ICSER Conference Defining and Evaluating ‘Science for Sustainability’ AIRP-SD Project Paul Weaver & Leo Jansen

NZSSES/ICSER Conference AIRP-SD Goals Develop an approach for ex ante, concomitant and ex post evaluation of sustainability-oriented RTD programmes Describe innovative RTD approaches Identify good and bad practices Assess transferability of practice Establish programme design guidelines

NZSSES/ICSER Conference Toward an evaluation architecture Lack of theory and practice suggests need to develop both through an iterative, heuristic procedure –begin with tentative hypotheses about the nature of the research processes that might support SD; test/refine these through the evaluation process itself Need for externally-specified evaluation criteria Depends on establishing a conceptual model of sustainability-oriented RTD

NZSSES/ICSER Conference AIRP-SD Procedure Establish conceptual model of sustainability research Select case study research programmes/projects Describe research outcomes, designs, contexts Evaluate outcomes, design, context separately Relate research outcomes to design Relate design to context Relate outcomes to context Revise conceptual model Recommendations for research management and design Identify good practice

NZSSES/ICSER Conference Derive conceptual model from: the nature and concept of sustainable development the nature of the socio-ecological systems of interest the nature of SD challenges to science and society

NZSSES/ICSER Conference SD challenges Orientation and ambition - paradigm change - transition - prescriptive - normative - vision-led Complexity of socio-ecological systems of interest, indeterminacy, irreversibility, risk, precaution Multiple stakeholders, high stakes, uncertainty, non-market values Need to communicate with non-scientists

NZSSES/ICSER Conference Evaluation objects (outcomes) Outcomes Contribution to strategic orientation Contribution to solution set (quantity/quality) Contribution to strategic management Contribution to social capital Contribution to scientific capital and capacities Contribution to influencing society

NZSSES/ICSER Conference Evaluation objects (D&P) Design and Process Management and finance model Procedure for initiating/defining programme Research model (methods, tools, procedures) Procedure for handling risk and uncertainty Procedure and basis for decision making Procedure for communication & dissemination

NZSSES/ICSER Conference Evaluation objects (context) Context Established social capital Established RTD management capital Established scientific/research capital Extent & urgency of unsustainability or vulnerability Understanding & awareness of (un) sustainability across society

NZSSES/ICSER Conference Sustainability principles Global and systemic approaches Knowledge sharing and mutual learning Broad participation Environmental heritage Precaution / Action to prevent risk Resilience

NZSSES/ICSER Conference Sustainability principles (cont.) Transparency and justification of decision making Appropriate scale-issue relation Inter-generational equity Intra-generational equity

NZSSES/ICSER Conference AIRP-SD working hypotheses Hypothesis I A wide set of generic research outcomes can be identified and evaluated on a set of externally specified sustainability goals and criteria Hypothesis II Research designs and processes that conform strongly with the principles of sustainable development are likely to contribute to strong research outcomes Hypothesis III Contexts where sustainability problems or awareness is high are likely to facilitate SD research. Strong scientific/social capital in relation to SD challenges will facilitate SD research and strong outcomes

NZSSES/ICSER Conference Hypothesis IV CONTEXT Research design and process Research outcomes Research context, design/process and outcomes are connected in a loop

NZSSES/ICSER Conference Nine case studies ALR ATSD CUTS ECO-CYCLE HABIFORUM MAHRE SEP STD ZERI

NZSSES/ICSER Conference Fact finding versus evaluation Separate tasks Data collection – factual information – What? How? Who? Evaluation – analytical, judgmental – Whether? How well? Evaluative questions shape the mindset and background for fact-finding

NZSSES/ICSER Conference Methodological issues Restricted time-frame for analysis; some outcomes yet to emerge: interim outcomes, process-orientation Ambiguities; e.g., financing model: detailed specification and differentiation Subjectivity: consistent definitions, inter- subjectivity and cross-checking Depth vs breadth: encourage evaluation to enable validation in due course

NZSSES/ICSER Conference Multi-level approach (outcomes)

NZSSES/ICSER Conference Evaluation of each object Evaluate each outcome using SD goals Evaluate each D&P feature using SD principles Evaluate each context feature using SD challenges Strength (strong, weak, neutral) Direction (positive, negative)

NZSSES/ICSER Conference Evaluation of interrelations Evaluate influence of each D&P feature on outcomes Evaluate influence of each D&P feature on each other (synergies) Evaluate influence of context (context dependency, transferability) of each D&P feature Evaluate any RTD induced change in context

NZSSES/ICSER Conference Data analysis

NZSSES/ICSER Conference Results: good practices Initiation –stakeholder inventory/analysis –broad participation –re-conceptualisation, scope/goals Finance –co-funding arrangements –dedicated funds for re-conceptualisation, stakeholder participation, communication –phased funding arrangements

NZSSES/ICSER Conference Results: good practices (cont.) Management –internal learning (iterative, adaptive) –openness, transparency, accountability –continuous monitoring and quality assurance –re-defined set of success criteria Research –vision-led, system-based, long-term oriented, inter/trans-disciplinary, iterative –multi-functional approaches/tools

NZSSES/ICSER Conference Conclusions Importance of context Importance of structure and process Importance of goal setting Importance of SD principles Synergy among design features - need for thorough integration Feedback - practice makes perfect

NZSSES/ICSER Conference Defining and Evaluating ‘Science for Sustainability’ AIRP-SD Project Paul Weaver & Leo Jansen