Measurable, Meaningful, and Motivational Metrics Miriam L. Goldberg AESP WI Chapter November 5, 2008

Comments on Peters-McRae Provocative, thoughtful, provides useful insights on how we assess markets The case against “net” is overstated “Current FR estimation approaches” is a straw man, weaker than most actual practice There are biases both directions in end-user self-report methods Greater interest in environment & energy efficiency means actual free ridership is likely to be up Not just apparent “social desirability” Good design doesn’t aim for minimum FR It aims for high cost-effectiveness

Some principles Using public money requires evidence that we’ve accomplished something that wouldn’t have happened without it. That is, we need net, not just gross savings What gets accomplished depends on what gets rewarded. What gets rewarded depends on what gets measured, and how. Measurement matters

Options Let what we can measure rigorously drive policy goals & program design Set policy we want regardless of measurement capability Work from both these extremes to an acceptable middle ground Need to balance the ideal and the practical for both policy and measurement

Why Measure Attribution? Cost-Effectiveness assessment Determining net program value Not just go/no-go, but how does this use of public money compare with other possible uses Portfolio design and resource allocation Program design/improvement Tracking against (net) goals Determine incentive payments Increase accomplishments Requires net Doesn’t require net

Problem: Social research is imperfect Response: We know more than nothing Programs where end-user self-reports can’t give useful information are only a part of today’s portfolios Regions new to DSM start with basic designs Many programs will never yield large market effects For programs that affect suppliers & multiple market dimensions over time, we have many other tools Imperfect measurement can still be useful for tracking improvement over time & guiding decisions

Problem: Asking operating programs to limit FR puts sales teams in a strange bind. Sell only to customers who don’t want your product much Response: Design programs to move markets to higher efficiency adoption, based on good market info Measure and reward program accomplishment based on gross savings Leave program operators free to pursue customers within the program design Cost-effectiveness, program re-design, portfolio re-allocation still require attribution assessment

Problem: Emphasis on short-term attribution discourages investment in long-term market changes and broader efficiency improvements Response: Understanding program effectiveness requires both short- & long-term perspectives If today’s free riders are yesterday’s spillover That doesn’t make today’s program still worth running Mechanisms for quantifying and rewarding delayed effects ideally should be built into original program Creating lasting change in markets is difficult

Dynamic Baseline A Framework for Planning and Assessing Publicly Funded Energy Efficiency, C 7, PG&E 2001

A Modest Proposal: Moving Beyond Widget Tracking Set regional energy use targets Possibly functions of population, production, weather, economic conditions Design programs to lower regional energy use Measure and reward success based on regional indicators Use market studies to figure out what programs are working and how to fine-tune design parameters Move the attribution argument to understanding how to make things work better, not fighting for payments