Experience you can trust. Market Transformation Programs: What they are. How to evaluate them. Kansas Corporation Commission Mitchell Rosenberg, Vice President Topeka, Kansas March 26, 2008

2 Overview Market Transformation: Definitions and Reality Importance of MT Concepts in Energy Efficiency Program Design and Evaluation Key Challenges in MT Evaluation Steps in Meeting MT Evaluation Challenges: examples from actual programs & studies Lessons Learned

3 Definition of Market Transformation Market Effect: a change in the structure or functioning of a market or the behavior of participants in a market that results from one or more energy efficiency program efforts. – Typically, these efforts are designed to increase the adoption of energy-efficient products, services, or practices, and are causally related to market interventions. Market Transformation: Market effects that persist once supporting programs are terminated

4 MT References the Product Life Cycle

5 Policy & Programs to Accelerate MT

6 EE Programs do help transform markets: The case of electronic ballasts Programs Underway

7 The case of resource-efficient clothes washers

8 Why MT Concepts are Important in Program Design & Evaluation Program designs must be matched to current stage of development to be cost effective Harnessing supply side motivations enhances cost effectiveness High GHG reduction goals – Need continuous pipeline of energy-efficient products, services, design practices – Need to extend adoption of established products to all relevant market segments In some cases, market research provides most fruitful approach to NTG evaluation

9 Key Challenges in MT Evaluation Effects occur over a long time frame – Events of interest span decades – Program planning & evaluation cycles typically 3 years or less Multiple program sponsors and policy makers are involved – Often coordinate activities among jurisdictions – How tease out effects of one states programs? Data are difficult to obtain – Manufacturer shipment data often proprietary – Distributors, contractors, retailers, designers tend to be small, scattered, disorganized

10 Steps in Addressing the Challenges of MT Evaluation Market Assessment/Program Logic Model Program History Development: Market Presence Indicators Selection of analysis objectives and strategy (ies) Appraisal of results/Adjustments for next rounds Close parallels to steps in program design and revision

11 Program Logic Model: Home Performance

12 Market Assessment/Program Theory Key Objectives – Location on the product life cycle – Identification of key market actors & segments, motivations, barriers – Assess data availability for program management and evaluation – Get program planners, implementers, regulators on same page re: short & long term goals and expectations Timing: Should begin in program design phase Resources: Avoid over-investment – Secondary data & reports useful – Selected local primary research often very useful

13 Program History/Market Presence Important to build reference for market effects and attribution analysis Should include national as well as local events & trends over time Data to capture – Program activities: marketing, training, incentives – Participation: customer and vendor, numbers and characteristics of participants and measures, timing, region – Efficiency levels supported v. baseline practice

14 Example: EVT clothes washer programs

15 Analysis Strategies Market Effects Indicators – Market share: sales, prevalence of practices, fleet efficiency indicators (e.g. mean EERs, MPG) – Price trends – Changes in codes and standards Attribution Analysis – Cross sectional – Time Series – Self-reporting – Historical Best to attempt multiple indicators and attribution analysis approaches

16 Net Effects of Vermont CW programs: Cross Sectional & Historical Approach Results of cross-sectional modeling – Dependent variable: State level Energy Star CW market share – Independent variables: Program indicators Customer Demographics & Energy Prices Change indicator

17 VT clothes washer programs: Whats happening to local program influence Methods – No-program Area market shares rising faster than market areas Market conditions – Impending federal minimum standard changes – Profitable product for manufacturers Fair representation of local program effects?

18 CA commercial lighting programs: presence in the market

19 General Challenges from C&I Programs Little available market share data Complex measures Greater customer market segmentation Multiple levels of supply chain with direct influence on project-level equipment selection and design Multiple decision makers and criteria in one customer Generally greater reliance on weight of evidence and self-reported approaches than in residential.

20 Other Potential Approaches to Attribution or Baseline Development Diffusion modeling Delphi or other expert judging Conjoint analysis of preferences for efficient substitutes for current products

21 Lessons Learned Generate as detailed a story as possible – Know the history of the program and its relatives – Know the history of the market and technology – Know what other programs are doing – Develop the program logic with local stakeholders Know the available data resources – Sampling, contents, collection methods – Know what others have done Design data collection to the attribution strategy or strategies

22 Lessons Learned: Maximizing data opportunities Quality of Indicators – Accuracy, face validity, bias (lack of same) Replicability (ability to support historical or time series analysis) – Sample frame: captures full population, updated regularly, documented compilation – Sampling: keep it kosher and document it Comparability (supports cross-sectional analysis) – Same definitions as data collected elsewhere – Capitalize on channels to reach national markets

23 The Bigger Picture Play well with others – Markets addressed by local programs are national and international – Many program operators are heavily involved already – get to know them – Initiatives under way to procure sales & shipment data Independent program influence? – Many local programs already coordinate operations, or are developed in explicit reference to each other (e.g.) codes & standards – Why try to tease apart effects?