Challenges & Strategies for Success Challenges & Strategies for Success Lewe Atkinson Manager, Evaluation and program improvement Evaluator as change agent
Summary What is MLA? Rationale for evaluation at MLA What is the MLA evaluation framework? Strategic framework for change Major challenges for change Successful change tactics Key points When will we know that we have been successful?
What is MLA? Industry-owned company Formed in 1998 Australian red meat industry Marketing, Market access & R&D H/O in Sydney others interstate 7 offices OS (US, Kor, Jap, China, Brussels, Russia, & Bahrain) 241 staff Expenditure about $160M/yr
Rationale for evaluation at MLA Effectiveness = doing the right things Efficiency = doing them well Benefit = doing the right things well Commonwealth stakeholders = matching $ and alignment with NRRPs Industry stakeholders = MLA accountability for levy income
Framework for independent program evaluation
What is the MLA evaluation framework ? Based on the DOFA framework. Input/output/outcome program logic framework. Reports alignment with Rural Research and Development Priorities. Consistent with the Council of Rural Research and Development Corporation “Guidelines for Evaluation”. Three major MLA programs undergo ex-post evaluation each year.
Strategic framework for change People-focused program logic: Jess Dart Brisbane MLA goal = build stakeholder confidence in MLA. 2. End-users = politicians, industry peak councils, and MLA board. 3. Next-users = MLA Staff: MD, EC, & PM’s. 4. We will be successful when; – Evaluations are meaningful to end-users – MLA staff manage for outcomes – Evaluation is core to MLA values
Major challenges for change 1. Making the complex simple 2. Embedding a focus on outcomes 3. Maintaining independence & consistency across programs 4. Building evaluation capability 5. Developing a learning culture
Successful change tactics 1. Alignment with MLA values, vision, and strategy 2. Engaging key stakeholders in setting better outcome KPIs 3. Engaging the Executive Committee 4. Creating cross functional teams, learning by doing & then reflecting 5. Creating layers of independence & transparency
Asking key evaluation questions Linking evaluation to strategy 1.Making the complex simple: Alignment with MLA values, vision, and strategy
What are the key evaluation questions…?
1.Are MLA programs adding value? 2.Are our collaborations efficient and effective? 3.Do we facilitate innovation from concept through to commercialisation? 4.Do we facilitate increased industry capability and investment in innovation? 5.Are we identifying and effectively managing industry risks?
Linking evaluation to strategy?
MLA strategic plan and evaluation
Program evaluation strategy and schedule
Link to MLA strategy
2.Embedding a focus on outcomes: Outcome KPI setting workshops across MLA & with key stakeholders Linking outcomes to employee performance measures Engaging key stakeholders in setting better outcome KPIs
Program evaluation is just the tip of the iceberg
Program logic encourages critical thinking Objectives Inputs Outputs Outcomes Market returns Benefit cost ratio Triple bottom line KPIs SocialEnvironmental Integrated framework
3.Maintaining independence & consistency across programs: Executive learning on the job – Sign-off on the MLA Evaluation Strategy Engaging the Executive Committee
Executive learning on the job 1. Annually determines a minimum of three major MLA programs to undergo an ex-post evaluation. 2. Must approve each program evaluation proposal prior to project commencement; – Scope – Methodology – Team – Budget – Timelines 3. Must approve each brochure in the evaluation series prior to publication.
4.Building evaluation capability: Creating an evaluation-focused organisational culture Creating cross functional teams, learning by doing & then reflecting
Living & learning MLA values workshop series
Monthly review of ex-post program evaluation series Last Friday of each month Theme based on one of MLA’s values and an expert presentation on a related evaluation topic Panel discussion: Economic benefit analysis reinforced by real world case studies Reflection on lessons learned to encourage a culture of continuous improvement
5.Developing a learning culture: Transparency – access via MLA website MLA contribution to RDC aggregate results Collaborating with evaluation experts to address the challenges for measuring MLA’s value add Creating layers of independence & transparency
Transparency – access via MLA website What will you find? – A message from the Chairman – Downloadable PDF version of annual evaluation schedule – Downloadable PDF overview of MLA evaluation framework – Downloadable PDF for each brochure in impact series – Archive of all source documents and references sighted in brochure series
MLA contribution to RDC aggregate results INPUT FROM MLA EVALUATION FRAMEWORK JOINT INPUT INTO COLLABORATIVE RDC PROGRAMS
Challenges for measuring MLA’s value add Credible counterfactual case – independent verification of MLA’s value add Realistic attribution – only outcomes under control of MLA Monitoring adoption – meaningful outcome KPIs, other indicators identified early and monitored Increasing program complexity – requires consistent application of program logic
Independent Consistent Transparent Corporate learning Continuous improvement Key points
When will we know that we have been successful? Evaluations are meaningful to end- users MLA staff manage for outcomes Evaluation is core to MLA values
Questions