Defining and measuring outcomes Definitions Workshop 11 Dec 2001 SSC

Purpose of Workshop Objective is to approve outcome framework as basis for writing BB1 document This includes basic definitions, etc (table)

Agreements Sought Agree to accept definitions in table, modified as discussed in this WS Approve the commencement of writing by Secretariat

Criteria for change Concepts first, then detail Table is a base, we seek improvement Majority agreement on what to change Compatible with NZ public sector management framework Best solution for New Zealand

Workshop Process 9:00-9:10 Welcome & Intro 9:10-10:00 Concepts of framework (three minutes, max) Each agency provides comments on concepts (rather than wording) 10:00-10:15 Group decision on concepts 10:15-10:30 BREAK 10:30-11:00 Concept by concept discussion of framework wording 11:00-11:30 Seek group agreement for secretariat to commence writing chapter 1

CONCEPT End Outcomes State Indicators Impact Measures Written definition of benefits from agency’s services as experienced by target group(s) in community State Indicators Quantitative* measures used to monitor changes in End Outcome between groups / over time Impact Measures Quantitative* measures of change in State Indicator caused by one or more activities of agency (or agencies) * Includes qualitative measures where data limitations / costs prohibit more objective measurement

Agency Outcome Definition (formerly ‘End Outcome’) Name / Definition Agency Outcome Definition: Definition in measurable terms of the impacts on, or consequences for, the community of the outputs or activities of agency / ies (PFA) Attributes Consistent, verbal definition of changes attributable to activities of one or more agencies Clearly identifies consequence of agency’s outputs or activities to the community Clearly referenced to target of services Definition is stable over time, and based on accessible source data Only defines how performance will be measured Limitations Limited use in decision-making or prioritisation

State Indicator Name / Definition Attributes Limitations State Indicator: Quantitative* measure of actual outcome for specified groups, based on Agency Outcome Definition Attributes Quantitative measure: clearly linked to agency services clearly linked to service recipients segmented by e.g. population groups, regions, business units, etc SMART/FABRIC (see NAO – March 2001, including: affordable; benefits outweigh measurement costs) Data obtainable within acceptable timeframe Statistically robust Limitations Unknown attribution to outputs or activities of agency, due to external influences Measurement errors

Impact Measure Name / Definition Attributes Limitations Impact Measure: Quantitative* measure of the change in a State Indicator caused by one or more activities of agency (or agencies) Attributes Same attributes as State Indicator, plus Compares State Indicators for ‘treated’ group with the reference group not receiving the treatment When used to compare different interventions or treatment groups, must measure impact using a consistent method Limitations Model robustness Ethical constraints on reference group(s) Measurement errors

Proposed BB1 Structure Target audience(s)? Structure and flow? Content?

Agreements Sought Agree to accept definitions in table, modified as discussed in this WS Approve the commencement of writing by Secretariat