1. Marjorie McRae, Ph.D., Principal Jane S. Peters, Ph.D., President Research Into Action, Inc., 503.287.9136

2. Energy Efficiency Programs Programs that encourage people to:  Replace inefficient equipment prior to equipment failure  Purchase energy-efficient equipment  Operate equipment to satisfy human and process needs with minimal energy use  Maintain equipment at maximum efficiency  Build energy-efficient spaces (including renovations, remodels, and tenant improvements)

3. Continuous Improvement in Managing Energy Efficiency Programs: …Is considered a best practice …Necessitates recurring process evaluation …Has been embraced by a few pioneering organizations …Has been thwarted by short program cycles and utility regulatory requirements …Has promise for states embarking on energy efficiency to meet long-term energy use reduction goals

4. Continuous Improvement as a Best Practice

5. Continuous Improvement Necessitates Recurring Process Evaluation Questions relevant at pre-launch and within first six months:  What are the measurable goals?  Does the program logic support goal attainment?  Does data collection support measuring goal attainment?  Is the program being implemented as designed?  Any implementation difficulties?  Are key actors responding as anticipated?

6. Continuous Improvement Necessitates Recurring Process Evaluation cont. Questions relevant at one year:  What are the goal measurements; is program on track to attain longer term goals?  What are the process steps; how long do they take; how many people are involved?  What are the “sticking points”, opportunities for improving efficiency?  What are the successes?  What are experiences of key actors?  What are the opportunities for improving effectiveness?

7. Continuous Improvement Necessitates Recurring Process Evaluation cont. Questions relevant in subsequent years:  All the questions at previous junctions may be relevant, as indicated by the previous findings.  In-depth looks at:  Adequacy of assumptions about market response  Reasons for nonparticipation among end-users and trade allies  Barriers to repeat participation and larger projects among end-users & trade allies  Need for new interventions (training, marketing, networking, etc.)

8. CI Can Benefit from Process Evaluations Across Programs Identify themes for program processes  Problems or opportunities common to multiple programs Better understand markets  End-users can be targeted by multiple programs  Trade allies can support multiple programs

9. Continuous Improvement Through Process Evaluation Has Been Embraced by a Few Pioneering Organizations  Northwest Energy Efficiency Alliance (NEEA)  New York State Energy Research and Development Authority (NYSERDA)  Energy Trust of Oregon

10. Characteristics of Organizations Pursing CI Through Process Evaluations Management commitment to CI and its concomitant self-appraisal Organizations geared to fundamental market change (“market transformation”) Long program cycles—five and ten years Freedom from typical regulatory constraints faced by utilities

11. NEEA accepted a proposal of Northwest Energy Efficiency Council (NEEC) to develop the BOC Intention: to make the BOC self-sustaining Program trains and certifies building operators in methods to attain energy-efficient high performance We conducted seven evaluations over five years

12. Building Operator Certification: Continuous Improvement Early evaluations addressed:  Quality of course content and teaching methods  Value of certification Middle evaluations addressed:  Size of market  Appropriate price for the training  Exploration of methods to estimate energy savings Later evaluations addressed:  Business model for self-sustaining  Estimation of average energy savings per participant

13. Intention: contribute to the development of a sustainable market for PV technologies Program elements:  Incentives for PV installations  Requirement to use “eligible” installers  Support for the development of accredited PV training programs  Promoting and facilitating nationally recognized certification for PV installers  Providing business development and market support incentives for PV dealers and installers

14. Photovoltaic Program: Continuous Improvement First evaluation addressed:  Program process steps and timelines from perspectives of all parties  Satisfaction of installers and end-users Second evaluation addressed:  Response to a NYSERDA-sponsored RE/EE workforce education conference Third evaluation addressed:  Relationship between PV installer workforce development activities and PV system outcomes

15. Intention: To increase the energy efficiency of industrial processes Program elements:  Incentives for energy efficiency upgrades to industrial equipment and processes  Technical assistance from industrial process specialists  Dedicated marketing/customer support staff to develop long-term relationships and assist with program processes

16. Production Efficiency Program: Continuous Improvement Sixth-month evaluation addressed:  Use of implementation contractor, with subcontractors, for program delivery  Quality of projects; energy impacts First-year evaluation addressed:  Follow-up on problems identified in 1 st evaluation  Stimulation of process improvements  Progress toward three-year goals Subsequent evaluations addressed:  Appropriateness of contracting approach  Savings realization rates by technology type

17. Barriers to Continuous Improvement Evaluation Short program cycles – 1 to 3 years –focuses evaluations on outcomes, not processes Lack of organizational commitment to energy efficiency and its continuous improvement Regulatory indifference Regulatory requirements focusing on outcomes Regulatory directives on appropriate topics for evaluation (“micro-managing”)

18. Continuous Improvement Evaluation Has Promise Many states are embarking on energy efficiency to meet long-term (10 year) energy use reduction goals  Michigan, Maryland, Ohio, Illinois, … Aggressive goals go beyond “low hanging fruit;” might not be attained without continuous improvement Regulators need to provide encouragement and flexibility

19. Evaluators Can Educate Regulators and Utilities Evaluators can educate regulators and utilities on:  Importance of continuous improvement for energy efficiency programs  Role of process evaluations in continuous improvement Education through  Professional organizations  Publications  Public forums, public comment periods

20. Continuous Improvement in Energy Efficiency Marjorie McRae, Ph.D., Principal marjorie@researchintoaction.com marjorie@researchintoaction.com Jane S. Peters, Ph.D., President Janep@researchintoaction.com Research Into Action, Inc. 503.287.9136