1. EM&V Framework Refresh Needs Assessment
Preliminary Interview Findings
Jane Peters, Research Into Action
Todd Malinick, Research Into Action
Kevin Cooney, Navigant Consulting
Greg Wikler, Navigant Consulting
Emrah Ozkaya, Southern California Edison

2. Agenda Policy Changes that Affect Evaluation Environment
Background, Objectives & Methods
Uses of the Framework
Policy Changes that Affect Evaluation Environment
New & Emerging Methods and Data

3. Study Background, Objectives and Methods

4. Study Background
The California Evaluation Framework was last revised in January 2006g11 years old
In past several years, policy decisions and legislation in California (e.g., AB802, SB350, etc.) have the potential to affect the Framework
Goal: Identify whether there are ways to improve the usefulness and usability of the Framework, especially in light of recent policy decisions and legislation

5. Research Objectives Uses of the Framework
How the Framework is used, who uses it, why it is and is not used and who should use it?
Can the Framework be more useful, more relevant, more available, or more user friendly for existing and new users?
Policy Changes that Affect Evaluation Environment
What are the effects of recent policy changes on evaluation?
What are the challenges for doing embedded evaluation under the current Framework and what guidance might be helpful for programs and processes?
Does the Framework have sufficient flexibility for other types of evaluations?
Is the Framework sufficiently flexible to allow for changes to cost effectiveness provisions?
New and Emerging Methods and Data
Are there useful or emerging methods that the Framework does not address?
Are there technologies that are not addressed in the Framework that should be considered for evaluation of EE/DSM/IDSM?

6. Methods
Conducted 44 of 45 targeted telephone interviews in May-June 2017
IOUs = 3
CCAs/RENs/LGPs = 8
Regulators = 4
Evaluation Firms = 10
Third-Party Implementers = 11
Data Analytics Vendors = 4
Other Parties = 4
Conducted comprehensive policy review
Conducted workshop at CEEIC Symposium
Overall Response Rate = 63%

7. Uses of Framework

8. Awareness of Framework is high…
Majority of respondents have heard of or used Framework
Greatest awareness among IOUs and EM&V firms
Some stakeholders have low awareness

9. …familiarity with & referencing of Framework are low
Most respondents have only read parts of the Framework; very few have read all of it…
… and most respondents indicated they refer to it “rarely” or “never”

10. Framework may be losing relevance
Respondents commonly experience situations not Addressed by Framework; no respondents said “Rarely” or “Never”
Respondents generally feel the Framework is not helpful for addressing measure/program performance or goal improvement or baseline conditions

11. Policy Changes that Affect Evaluation Environment

12. New policies and legislation may drive updates
New policy or legislative-driven topics warrant the most attention
Distributed energy & storage, water, and ancillary services of less interest

13. New and Emerging Methods and Data

14. Consensus on evaluation quality and timeliness
Respondents were asked to rank the timeliness, cost efficiency and robustness/quality of evaluations
Relative rankings are the same across all groups:
g Robust/Quality
g Timely
g Cost Efficient

15. Addressing embedded evaluation is critical
Most respondents familiar with embedded evaluation…
…and vast majority of respondents feel it is “very important” or “important” that embedded evaluation be addressed in the framework.

16. Most doing evaluation have designed experiment, but don’t look to Framework for guidance
Nearly all IOUs, data vendors, and evaluation firms have used experiments
Few implementers have
Very few stakeholders go to the Framework for guidance in designing experiments

17. Some have prepared evaluation report, but few look to Framework for reporting guidelines
As one would expect, IOUs and evaluation firms prepare and oversee evaluation reports
But few look to Framework for reporting guidelines

18. Some study types warrant new attention
Significant interest in most study types, though low income programs stand out
Less intertest in market research guidance

19. More to come… This covers only some of the preliminary results
Research Into Action and Navigant will be working on the remaining analyses and draft report this month
Expect the final report and public presentation in late July