1. MARKETING, EDUCATION AND OUTREACH (ME&O) EVALUATION FINDINGS CPUC ME&O Workshop April 14, 2016

2. Structure of Today’s EM&V Presentation  Statewide ME&O Evaluation Results & Stakeholder Discussion  Cross-Cutting ME&O Evaluation Results & Stakeholder Discussion CPUC ME&O Workshop - Evaluation Results 2 ME&O Administrator ProcessVerificationEffectiveness CSECross-Cutting SW Verification and Effectiveness IOUCross-Cutting RENCross-Cutting

3. Statewide ME&O Evaluation Findings CPUC ME&O Workshop - Evaluation Results 3

4. Program and Evaluation Overview CPUC ME&O Workshop - Evaluation Results 4

5. Program Overview CPUC ME&O Workshop - Evaluation Results  The SW ME&O program is a social marketing campaign implemented under the brand of Energy Upgrade California designed to educate, activate, and motivate Californians to take energy-saving actions.  The Center for Sustainable Energy (CSE) used a range of marketing channels to engage residential customers and meet program objectives  The total budget was $42.8 million for 2014-2015  The CPUC did not finalize performance metrics until May 2015 Source: CSE 5

6. Program Objectives CPUC ME&O Workshop - Evaluation Results 6  The CPUC outlined nine objectives for the SW ME&O program Educate on Programs, Energy Use, and Opportunities Encourage Engagement with Tools/Resources Communicate Benefits of Taking Action Direct Consumers on How to Take Action Identify and Pilot Messaging for Partners Identify and Pilots Methods for Small Business Develop Social Marketing Campaign Coordinate on Efforts, Messaging and Tactics Develop EM&V Roadmap

7. Program Metrics CPUC ME&O Workshop - Evaluation Results 7  The CPUC, CSE, and stakeholders participated in a process to develop specific metrics by which to judge performance of the program  These metrics are limited in scope, but represent a starting point from which to assess effectiveness Metrics Awareness of Energy Upgrade California Knowledge among IOU ratepayers who are aware of Energy Upgrade California of the specific actions and opportunities communicated by the initiative that they can take to better manage their energy use Engagement with Energy Upgrade California website, digital media, social media, and community outreach Participation in and engagement with Energy Upgrade California by CBOs, local governments, retailers, and realtors Small business messaging is researched and piloted RENs and IOUs provide information to CSE and the marketing firm in a timely manner EM&V roadmap for Energy Upgrade California is completed

8. Evaluation Overview: Audit and Verification Study Component CPUC ME&O Workshop - Evaluation Results 8 Evaluation ActivityEvaluation Purpose In-Depth Interviews with Program Staff Provides context around overall strategy and implementation of program activities Program Material ReviewServes as documentation of program activities Observations of Retail and CBO Events Provides qualitative data on the implementation of retail and CBO events, including the consumer experience In-Depth Interviews with CBOs Provides qualitative data on the role of CBOs in the program, their feedback on the process, challenges and recommendations for improvement

9. Evaluation Overview: Effectiveness Study Component CPUC ME&O Workshop - Evaluation Results 9 Evaluation ActivityEvaluation Purpose Residential General Population Surveys Provides a measure of change over time for key metrics among the general population Self-Report Attribution Survey Provides a measure of attribution to the program for key metrics such as awareness, knowledge and action taking Event Follow-Up Survey Mobile Outreach Survey

10. High-Level Findings CPUC ME&O Workshop - Evaluation Results 10

11. Multiple channels used to reach vastly different numbers of Californians CPUC ME&O Workshop - Evaluation Results Data Source: CSE Tracking Data 11 Shift in strategy to focus on one-on-one interactions

12. The program achieved most formal program performance metrics CPUC ME&O Workshop - Evaluation Results  In terms of formal performance metrics, CSE achieved four of the five metrics for which they were directly responsible  Achieved targets for awareness and knowledge (Metrics 1 and 2) MetricDescriptionTargetActual Performance 1Awareness of Energy Upgrade California20% aware (aided)20% 2 Knowledge among IOU ratepayers who are aware of Energy Upgrade California of the specific actions and opportunities communicated by the initiative that they can take to better manage their energy use 25% can identify highlighted programs (aided) 40% - 60% 25% can identify actions to save energy (unaided) 52% 25% know to go to the website to learn more (aided) a 43% 3 Engagement with Energy Upgrade California website, digital media, social media, and community outreach Website: 1.3 million unique visitors907,144 Website: 25% of visitors view ≥3 pages 21% Website: 30% of visitors spend >5 sec on a page 35% Social media: 40,000 Facebook fans 48,752 Digital media: 0.08% click through rate 0.11% 12

13. The program also achieved qualitative targets CPUC ME&O Workshop - Evaluation Results MetricDescriptionTargetActual Performance 4 Participation in and engagement with Energy Upgrade California by CBOs, local governments, retailers, and realtors Yes/NoYes - Achieved 5Small business messaging is researched and pilotedYes/NoYes - Achieved 6 RENs and IOUs provide information to CSE and the marketing firm in a timely manner Yes/NoYes – Achieved by IOUs/RENS 7 EM&V roadmap for Energy Upgrade California is completed Yes/NoYes – Achieved by CPUC 13

14. Additional key findings suggest overall performance is mixed CPUC ME&O Workshop - Evaluation Results  Given the limited scope of the program performance metrics, the evaluation team highlighted four additional areas pertinent to assessing effectiveness of the program:  Unaided Brand Awareness  Brand Familiarity and Knowledge  Energy Self-Efficacy  Energy Saving Action 14

15. Unaided awareness of the brand remains low CPUC ME&O Workshop - Evaluation Results  Consumers struggle to name any energy-related brands unaided Source: Wave 1 and Wave 2 Tracking Surveys. 15

16. Familiarity with the brand has increased significantly CPUC ME&O Workshop - Evaluation Results  It is possible that while aided brand awareness levels have not changed since the start of the program, the campaign may have increased people’s depth of knowledge or what people associate with the brand. How familiar are you with Energy Upgrade California? Jan 2013 (A) (n=374) May 2014 (B) (n=179) April 2015 (C) (n=125) Nov 2015 (D) (n=107) 1 – I have only heard the name32%16%31%17% 216%13%10%3% 313%15%9%5% 410%24%15%30% 515%9%17%22% 66%9%12%16% 7 – I know a lot about it7%15%6%7% MEAN3.09 BD 3.84 AC 3.38 BD 4.11 AC 16

17. Consumers show moderate levels of energy self-efficacy CPUC ME&O Workshop - Evaluation Results 17 Note: An asterisk indicates a statistically significant difference between subgroups at the 90% level. Source: Wave 1 and Wave 2 Tracking Surveys (EF1-EF9). High Self-Efficacy Low Self-Efficacy

18. Consumers interacted with the program in different ways CPUC ME&O Workshop - Evaluation Results  Consumers were significantly more likely to remember some types of interactions than others Note: Letters are assigned to each marketing channel. Letters next to percentage indicate the percentage is significantly different from the indicated marketing channel at the 90% level. Source: Attribution, Event Follow-Up, and Mobile Surveys, Fall 2015. 18

19. Changes in daily routine are more common than home improvements CPUC ME&O Workshop - Evaluation Results Note: Letters are assigned to each marketing channel. Letters next to percentage indicate the percentage is significantly different from the indicated marketing channel at the 90% level. Source: Event Follow-Up and Mobile Surveys, Fall 2015. 19

20. Recommendations for the RFP Process CPUC ME&O Workshop - Evaluation Results 20

21. Implications of Results for the RFP Process CPUC ME&O Workshop - Evaluation Results  Provide early and consistent direction on the objectives of the program  If objectives change, metrics should be adjusted  Establish more comprehensive program performance metrics  Establish metrics and KPIs that are appropriate for each channel  Document the link between ME&O activities and metrics  Engage the SW ME&O evaluator in the selection of metrics  Require bidders to outline how they will track program activities, KPIs, budget and spending in a centralized manner. In addition, comment on the frequency with which the data will be updated, how it will be made accessible and any QA/QC procedures they will use to ensure data accuracy 21

22. Implications of Results for the RFP Process CPUC ME&O Workshop - Evaluation Results 22  Pursue additional research strategies to address gaps in PY14-15 research  Ask bidders to comment on their ability to incorporate experimental design or other alternative implementation strategies  Success in this area requires close collaboration between implementation and evaluation teams

23. Cross-Cutting Evaluation Findings to Date CPUC ME&O Workshop - Evaluation Results 23

24. Evaluation Overview CPUC ME&O Workshop - Evaluation Results 24

25. Findings Drawn from Four of Seven Study Tasks CPUC ME&O Workshop - Evaluation Results 25  Interim deliverable provided on design and implementation issues  Recently completed interviews on topics of coordination and referrals ObjectiveArea of ExplorationPurpose Design and Implementation SW governance structure Document SW ME&O program design, as well as factors that influence design and governance, and whether the CPUC can provide more clarification around each stakeholder’s role. Budget allocation and tracking Document how and why funding earmarked for selected ME&O campaigns/programs is allocated, spent, and tracked. Campaign objectives and activities Document the goals and objectives of ME&O activities, and the specific activities implemented. Future program design Determine whether the CPUC should provide guidance to IOUs and RENs regarding their approach to marketing, as well as whether CSE and the IOUs/RENs should continue to provide a mix of messages about utility programs, no/low cost actions, etc. Coordination Coordination across ME&O administrators Document the ways in which ME&O administrators are currently coordinating between one another, assess how well that coordination is occurring, and identify potential opportunities for improvement. Referrals to IOU/REN programs Explore the barriers and benefits to developing and implementing a referral process from CSE to the IOUs and RENs, as well as whether, in the future, CSE should refer consumers to IOU and REN programs. Consumer Perspectives Consumer engagement and interaction with ME&O across administrators (the intersection of ME&O efforts) Explore how consumers engage with specific ME&O campaigns, their perceptions of the marketing they are exposed to (i.e., whether they resonate with them), potential market confusion, and the achievement of campaign objectives.

26. Design and Implementation Research Questions Focus on What Was Done and Why CPUC ME&O Workshop - Evaluation Results Budget Allocation & Tracking  What approach does each administrator use to identify how it will use its marketing dollars for a program/campaign?  How are ME&O budgets developed, and how are the budgets connected to program/campaign objectives and goals?  How were PY14 ME&O budgets allocated and why? Objectives and Activities  What ME&O activities did the IOUs and RENs conduct for target audiences in 2014? What was their purpose and strategy? What tactics/tools/channels were utilized?  What do the IOU and REN administrators track to assess the effectiveness of their ME&O activities? 26

27. Focused on Overall Approach to Strategy, as well as Specific Cases CPUC ME&O Workshop - Evaluation Results  With each program administrator (PA), the evaluation team explored:  Go-to-market strategy for 2013-2014 statewide residential EE programs  ME&O activities for specific programs including goals and objectives, budget, metrics and results:  BayREN – Home Upgrade  PG&E – Residential Energy Advisor and Home Upgrade  SCE – Residential Energy Advisor and Home Energy Efficiency Rebates  SCG – Plug Load and Appliances, and Energy Hero Campaign  SDG&E – Home Upgrade and Home Energy Advisor  SoCalREN – Home Upgrade and Green Building Labeling  While each PA is unique and administers ME&O differently, we present overarching take-aways from the research today 27

28. Coordination and Referrals Research Questions Document Efforts to Date and Future Opportunities CPUC ME&O Workshop - Evaluation Results Coordination  What are the goals and benefits of coordination?  How do administrators communicate and share information about SW ME&O activities?  How well is coordination occurring between administrators?  What coordination challenges and opportunities exist? Referrals  How does CSE currently refer interested customers to IOU or REN programs?  How do IOUs or RENs refer customers to each others’ programs?  What are the barriers and benefits to developing and implementing a referral process?  Should CSE refer customers to IOU programs? If so, what programs? 28

29. High-Level Findings CPUC ME&O Workshop - Evaluation Results 29

30. What approach does each administrator use to determine how it will deploy its marketing dollars for a program/campaign? CPUC ME&O Workshop - Evaluation Results  A multi-step process is used involving (1) the identification of program needs, (2) a determination of how marketing can support those needs, and (3) establishment of ME&O goals for the program cycle General ME&O Planning and Implementation Process 30

31. How are ME&O budgets developed, and how are the budgets connected to program/campaign objectives and goals? CPUC ME&O Workshop - Evaluation Results  Marketing plans translate ME&O goals into specific activities with budgets designed to achieve particular outcomes. Informed by various inputs including:  While all PAs develop marketing plans, the timing of their development and use varies across PAs, programs, and program cycles  PAs tend to develop more comprehensive and targeted plans for programs with larger energy savings or participation goals  Nearly all marketing plans provided campaign objectives, target audience, channel mix and timing. Fewer marketing plans provided key performance indicators (KPIs) or success criteria 31 Program business plans Past marketing performance Feedback from program staff

32. How were PY13-14 ME&O budgets allocated and why? CPUC ME&O Workshop - Evaluation Results  Budget allocation decisions are made based on the goal setting and marketing needs assessment process discussed above  Driven by target audience and approaches best suited to them  Among statewide residential EE programs, PAs allocated the largest ME&O budget to Energy Upgrade California Home Upgrade Program Program Name Total Statewide ME&O Budget % of Budget Energy Upgrade California Home Upgrade Program$11,831,28242% Plug Load and Appliances$8,951,36032% Residential Energy Advisor$3,735,41513% Residential New Construction$1,377,1435% Multifamily$1,155,8444% Residential HVAC$1,249,6804% Total$28,300,724100% 32

33. What ME&O activities did the IOUs and RENs conduct for target audiences in 2014? CPUC ME&O Workshop - Evaluation Results  The PAs used a wide range of marketing channels to reach their target audiences. ME&O activities might include: 33

34. What do the IOU and REN administrators track to assess the effectiveness of their ME&O activities? CPUC ME&O Workshop - Evaluation Results  Just over three quarters (107 of 141) of the PA ME&O activities reviewed for this task had KPIs  The RENs typically have the fewest KPIs, while the IOUs provide them more consistently  KPIs could be either quantitative or qualitative by activity  Quality of KPIs varied across PAs, programs and activities  Far fewer PAs have defined success criteria for their ME&O activities (32% overall; between 0% and 89% per PA)  PAs provided achievements for two thirds of their ME&O activities  In nearly all cases, the PAs do not report results that causally link an ME&O activity to a program outcome 34

35. How well is coordination occurring between administrators? CPUC ME&O Workshop - Evaluation Results  All stakeholders mentioned that coordination has generally improved as the various parties have worked together over time 35 Increased two- way communication More collaborative quarterly stakeholder meetings Earlier opportunities to provide feedback in the creative process Improvements to the TRUMBA marketing calendar

36. What coordination challenges and opportunities exist? CPUC ME&O Workshop - Evaluation Results 36  However, several coordination challenges remain, especially in terms of planning ME&O efforts Increased two- way communication Earlier opportunities to provide feedback in the creative process Prevents providing information in a timely manner Stakeholders develop plans in isolation before sharing with others Separate planning processes Prevents thorough feedback No time for input from other internal stakeholders in IOU/REN organizations IOUs/RENs do not always have enough lead-time for CSE requests Difficult to ensure that statewide ME&O efforts accurately reflect IOU and REN program details CSE does not receive timely updates on program changes Discourages feedback from the IOUs/RENs Makes IOUs and RENs feel their time and input is not valued CSE does not always acknowledge IOU/REN feedback when not used

37. Recommendations & Implications for the ME&O Planning Process CPUC ME&O Workshop - Evaluation Results 37

38. Recommendations Focused on Documentation and Data Tracking CPUC ME&O Workshop - Evaluation Results  Recommendations from this initial phase of research suggest that PAs implementing ME&O activities should:  Develop marketing plans for all programs or campaigns with clear objectives  Consistently develop KPIs and associated success criteria for their ME&O campaign and specific-activities  Determine how the achievement of success criteria will be assessed (i.e., quantitative or qualitative measurement)  Track spending on a per activity basis so that return on investment/cost effectiveness can be determined 38

39. Implications for the ME&O Planning Process CPUC ME&O Workshop - Evaluation Results 39  To enhance coordination and support future planning processes:  Build in lead-time for stakeholders to provide input on SW marketing efforts  Develop feedback loop for integration of stakeholder input  Brainstorm strategies for providing regular updates regarding programmatic changes to the SW implementer  PA business plans should include information to support coordination of ME&O efforts (e.g., identifying core target audiences)

40. Project Contacts CPUC ME&O Workshop - Evaluation Results  CPUC  Rory Cox - rory.cox@cpuc.ca.govrory.cox@cpuc.ca.gov  Opinion Dynamics  Tami Buhr – Roadmap Lead – tbuhr@opiniondynamics.comtbuhr@opiniondynamics.com  Hannah Arnold – Project Director – harnold@opiniondynamics.comharnold@opiniondynamics.com  Olivia Patterson – Cross Cutting Study Manager -- opatterson@opiniondynamics.com opatterson@opiniondynamics.com 40