1. 2015 Comprehensive LGP Process Evaluations Draft Research Plan with Common Scope of Work John Boroski and Tami Rasmussen, Evergreen Economics

2. Agenda 2 IOUs’ approach to staggered LGP Process Evaluations Overarching guidelines for process evaluation research LGPs being evaluated in Phase 1 Common scope of work: Objectives, Methods Research Team and Timeline Q&A, LGP information needs Next Steps

3. Process vs. Impact Evaluations 3 Energy Division conducts impact evaluations Impact evaluations identify and estimate energy and demand savings from a program. Program administrators may conduct process evaluations: Process evaluation is a systematic assessment of an energy efficiency program:  To document program operations at the time of the examination.  To identify improvements that can be made to increase the program’s efficiency or effectiveness for acquiring energy resources.

4. Comprehensive and Staggered LGP Process Evaluations “Comprehensive”: Every LGP activity is documented and evaluated “Targeted”: A subset of LGP activities are documented and evaluated. ‑ Research Into Action’s current process evaluation is targeted to municipal building retrofits and some Strategic Plan activities. The current effort is a comprehensive process evaluation. The last comprehensive process evaluations of LGPs were conducted during the 2006-2008 program cycle. LGP evaluations will be staggered in phases Work will be spread out over 4-5 years, depending on EM&V budget. Will achieve a census of comprehensive process evaluations. LGPs through multiple IOUs will be studied once. IOU comprehensive process evaluations will use a common SOW Ensures consistency across phases

5. Overarching Guidelines for Process Evaluation Research 5 1) Focus is on existing program activities only. 2) A logic model is a graphical mapping of key program activities, which are then linked to expected short-term outputs, and then longer-term outcomes. 3) The logic model process drives identification of program delivery needs and gaps, as well as innovative activities LGPs may be conducting. 4) Program implementation recommendations are only made when findings/data show there is an unmet program delivery need or gap – that is, outputs or outcomes are not occurring as planned.

6. Example Logic Model 6

7. LGPs in Phase 1 7 SCG/SCE:  Los Angeles County  Riverside County  San Bernardino County  Valley Innovative Energy Watch (VIEW, with PG&E) PG&E  Association of Monterey Bay Area Governments (AMBAG)  San Luis Obispo County (with SCG)  San Mateo County  Sierra Nevada SDG&E  City of Chula Vista

8. Common SOW Research Objectives 8 Document current LGP activities. Assess if each LGP is being implemented according to logic model.  Are any activities not occurring, or able to be improved? Document LGP successes and challenges. Gauge participant satisfaction with LGPs. Assess if LGPs are on track to meet CPUC- approved program objectives. Provide customized design/implementation recommendations for each LGP. ‑ Third level  Fourth level

9. Methods Overview Review and, if necessary, help each LGP update their logic model. Review program documentation and regulatory reports, including marketing collateral promoting core programs. Interview IOU program managers, support staff, third party implementers and vendors. Interview participating jurisdiction implementation staff and key stakeholders. Survey IOU customers/LGP participants. (Potentially) conduct other activities as needed to address researchable items.

10. Review/Update LGP Logic Models 10 Objectives: Understand and document program activities, expected outcomes. Inform data collection. Update models to reflect actual implementation. Approach/Issues: Do updates after IOU staff interviews completed, documents reviewed.

11. IOU Staff Interviews 11 Objectives (high level): Confirm current program activities. Assess implementation effectiveness, successes, challenges. Confirm outcomes from LGP activities. Solicit suggestions for program improvement. Obtain program materials, participant samples. Approach: IOUs provide contacts for LGP managers, supporting staff, implementation contractors. In-depth phone interviews lasting 60 to 90 minutes.

12. Government Partner Interviews 12 Objectives (high level): Document success and challenges related to program activities, activity gaps and needs. Confirm outcomes from LGP participation. Get opinions on IOU services, support, coordination. Solicit suggestions for program improvement. Approach: IOUs provide LG partner contacts – who identify key support staff. In-depth phone interviews lasting 60 to 90 minutes.  Follow up inquiries as needed

13. On-site Program Observations 13 Objectives (high level): Observe first hand how activities are conducted, services are delivered. Document delivery logistics, event effectiveness, stakeholder engagement. Document potential improvement areas. Approach: IOUs and government partners suggest events, meetings worth attending; IOUs approve. Evergreen spends 1-2 days on-site documenting activities, effectiveness, key findings.

14. Participant Surveys 14 Objectives (high level): Document sources of program offering awareness. Assess ease of program participation. Assess satisfaction with LGP services, measures. Solicit suggestions for improved participation. Assess propensity for future participation. Approach: Phone surveys lasting 15 minutes or less. Target completes = 50 to 100 per customer segment, per LGP (TBD after participant samples provided).

15. Notes on Data Collection Tools 15 Interview guides, phone survey instruments developed after Research Plan finalized. Evergreen will iterate with IOUs as needed to update and finalize draft guides and surveys. Will develop an analysis plan for quantitative surveys. Economies of scale assumed in RFP and work scope. Need IOU approvals of “core” instruments that can be replicated. May integrate a few LGP-specific questions.

16. Reporting 16 Objectives/Approach: Document everything the evaluators did and learned in one report for each IOU.  Separate sections for different LGPs Identify program processes that are working well or need improvement. Provide higher and lower priority recommendations that are actionable.  Who will do what, how and intended result Give public presentation on draft reports and integrate feedback.

17. Research Team 17

18. Evaluation Schedule TaskStartEndDeliverables Task 1A: Present Draft Research PlanJune 13, 2016June 24, 2016 Draft research plan posted, public webinar, meeting notes for SCG Task 1B: Review and Update Logic ModelsJuly 2016September 2016Updated logic models (after IOU staff interviews) Task 2: Revise Research PlanJune 27, 2016July 18, 2016 Final research plan posted, research issues for future LGP studies documented Task 3: Develop Interview Guides, Survey Instruments and Analysis Plan July 2016August 2016 Draft and final interview guides, draft and final survey instruments and analysis plans Task 4: IOU Program Manager InterviewsAugust 2016September 2016Completed interviews, disposition memo Task 4: Local Government Partner InterviewsAugust 2016October 2016Completed interviews, disposition memo Task 4: On-site Program ObservationsAugust 2016November 2016Event attendance, observations/findings memos; events TBD Task 4: Participant SurveysSeptember 2016November 2016Sampling plans, disposition reports, topline reports Task 5: ReportingOctober 2017December 2017Draft report template, completed draft reports Task 5: ReportingJanuary 2017 Revised draft report Task 5: ReportingTBD Public webinar Task 5: ReportingFebruary 2017 Final report with documented comments

19. Q&A and Suggestions 19 1.Questions on research objectives or work scope? 2.Other process evaluation research issues to consider? LGPs - What information do you need to help you implement your existing LGP activities more effectively?

20. Next Steps 20 Please submit written comments, suggestions on Draft Research Plan at: http://energydataweb.com/cpuc Search for “Comprehensive Process Evaluations of the PG&E, SCE, SCG and SDG&E Local Government Partnerships: First Wave.” Comments accepted through June 27. Evergreen and IOUs will review comments and develop final research plan with updated schedule. Same core research plan will be used for each IOU/LGP.

21. Next Steps 21 Note that: Research issues that are out of scope for the LGP process evaluations will be documented for the Joint EM&V Long Term Research Plan and considered for future studies. Upcoming ED-led LGP evaluation studies may be able to accommodate evaluation questions that do not address existing program activities.

22. Upcoming LGP Evaluations 22 -- From the 2013-2016 Energy Division & Program Administrator Energy Efficiency Evaluation, Measurement and Verification Plan, Version 6

23. More Questions? 23 John Boroski, Evergreen Economics boroski@evergreenecon.com