EM&V Framework Refresh Needs Assessment Preliminary Interview Findings Jane Peters, Research Into Action Todd Malinick, Research Into Action Kevin Cooney, Navigant Consulting Greg Wikler, Navigant Consulting Emrah Ozkaya, Southern California Edison

Agenda Policy Changes that Affect Evaluation Environment Background, Objectives & Methods Uses of the Framework Policy Changes that Affect Evaluation Environment New & Emerging Methods and Data

Study Background, Objectives and Methods

Study Background The California Evaluation Framework was last revised in January 2006g11 years old In past several years, policy decisions and legislation in California (e.g., AB802, SB350, etc.) have the potential to affect the Framework Goal: Identify whether there are ways to improve the usefulness and usability of the Framework, especially in light of recent policy decisions and legislation

Research Objectives Uses of the Framework How the Framework is used, who uses it, why it is and is not used and who should use it? Can the Framework be more useful, more relevant, more available, or more user friendly for existing and new users? Policy Changes that Affect Evaluation Environment What are the effects of recent policy changes on evaluation? What are the challenges for doing embedded evaluation under the current Framework and what guidance might be helpful for programs and processes? Does the Framework have sufficient flexibility for other types of evaluations? Is the Framework sufficiently flexible to allow for changes to cost effectiveness provisions? New and Emerging Methods and Data Are there useful or emerging methods that the Framework does not address? Are there technologies that are not addressed in the Framework that should be considered for evaluation of EE/DSM/IDSM?

Methods Conducted 44 of 45 targeted telephone interviews in May-June 2017 IOUs = 3 CCAs/RENs/LGPs = 8 Regulators = 4 Evaluation Firms = 10 Third-Party Implementers = 11 Data Analytics Vendors = 4 Other Parties = 4 Conducted comprehensive policy review Conducted workshop at CEEIC Symposium Overall Response Rate = 63%

Uses of Framework

Awareness of Framework is high… Majority of respondents have heard of or used Framework Greatest awareness among IOUs and EM&V firms Some stakeholders have low awareness

…familiarity with & referencing of Framework are low Most respondents have only read parts of the Framework; very few have read all of it… … and most respondents indicated they refer to it “rarely” or “never”

Framework may be losing relevance Respondents commonly experience situations not Addressed by Framework; no respondents said “Rarely” or “Never” Respondents generally feel the Framework is not helpful for addressing measure/program performance or goal improvement or baseline conditions

Policy Changes that Affect Evaluation Environment

New policies and legislation may drive updates New policy or legislative-driven topics warrant the most attention Distributed energy & storage, water, and ancillary services of less interest

New and Emerging Methods and Data

Consensus on evaluation quality and timeliness Respondents were asked to rank the timeliness, cost efficiency and robustness/quality of evaluations Relative rankings are the same across all groups: g Robust/Quality g Timely g Cost Efficient

Addressing embedded evaluation is critical Most respondents familiar with embedded evaluation… …and vast majority of respondents feel it is “very important” or “important” that embedded evaluation be addressed in the framework.

Most doing evaluation have designed experiment, but don’t look to Framework for guidance Nearly all IOUs, data vendors, and evaluation firms have used experiments Few implementers have Very few stakeholders go to the Framework for guidance in designing experiments

Some have prepared evaluation report, but few look to Framework for reporting guidelines As one would expect, IOUs and evaluation firms prepare and oversee evaluation reports But few look to Framework for reporting guidelines

Some study types warrant new attention Significant interest in most study types, though low income programs stand out Less intertest in market research guidance

More to come… This covers only some of the preliminary results Research Into Action and Navigant will be working on the remaining analyses and draft report this month Expect the final report and public presentation in late July

jane.peters@researchintoaction.com todd.malinick@researchintoaction.com kevin.cooney@navigant.com greg.wikler@navigant.com