Net-to-Gross: A Few Observations Vis-á-Vis the Long Term CALMAC San Francisco, California July 17, 2007 Michael Rufo Itron Inc Broadway, Suite 1800 Oakland, CA

Love ‘em, hate ‘em, but… –Free ridership and spillover/MT Important concepts, reasonably well understood, important to program design, forecasting/procurement, policy making But terms and measurements don’t always capture and convey what we want Free ridership and spillover/MT –Two sides of the same coin It’s short-sighted to focus on one while ignoring the other What matters is long term market change: –Totality of: Direct program participation, market effects, codes/standards, and other influences Free ridership is limited as a short-term, participant-driven metric Theory

Why Worry About Free Ridership? Non-participant’s perspective –Don’t give my money to someone else to do something they were doing anyway Program efficacy –Competition for scarce public purpose dollars Concerns can’t be cavalierly dismissed –and someone will always raise them! But… –when programs are effective over the long-term –and participation is widespread –concerns can be mitigated, if not eliminated

Program Effects are Often Acceleration (RER 2001)

Practice But…there are problems, mountains of them Adoption is non-linear and so is free ridership Markets are dynamic and when interventions succeed change is accelerated –Line AD in previous slide is not what we observe –What is a true naturally occurring baseline 25 years after the first market interventions? Today’s free riders are often yesterday’s market effects Measurement techniques are limited –No pure control groups Whatever happened to experimental design! –The vagaries of self reports –The pain of market tracking

Practice Measurement challenges partially related to under and inconsistent investment –~$1-2B/yr over past 15 years on programs How much on evaluation and longitudinal baselines? –Evaluation efforts spotty and often half hearted Real-learning the same inconclusive stuff over and over instead of conducting rigorous, consistent, and, yes, sometimes more costly, long-term research –No surprise we have very few reliable longitudinal data sets of market saturation, penetration, costs Nationally, reported program data is weak! –Doesn’t support econometric analysis

Extreme cases –Linear scaling of reward/penalty, threshold triggers –No financial feedback Nationally, some fallout from the1990s? –Fear and stipulation –Let’s call the whole thing off –Maybe if we don’t measure it it will go away But ex post NTG can provide vital feedback –Critical to improving program design –In some cases, partially reward/penalize depending on what administrators can realistically control –Not a substitute for multi-year market effects analyses NTG Application to Implementers

NTG Under Aggressive Program Funding Over the long term –EE program $ essentially savings and investment fund –Customers are really using their own money To enhance buying power Stimulate new markets Mitigate market barriers Explicit in “Use it or loss it” approach for large C&I –Short-term free ridership becomes less of a concern If long-term participation is widespread Significant market effects are accomplished Programs/policies adapt quickly to accomplishments/failures –Latter requires continuous measurement of both short- term program impacts and long-term market effects

Conclusions Not measuring should not be an option But traditional focus on “free riders” is suboptimal and can lead to wrong conclusions “Free rider” term itself problematic/derogatory –Not consistent with traditional economic use of term What matters are –Short-term “marginal program effectiveness (MPE)” Let’s find a better term that gets at what we care about –Long-term market effects And, yes, associated program attribution.

Conclusions Attribution is obviously challenging –Precise attribution will always be difficult –“Chunky” attribution less so Results should be used to –Optimize program and portfolio design Know when to –declare victory, admit failure, redesign, or move on –Appropriately direct and motivate implementers, w/out Penalizing for factors outside direct control Creating perverse incentives –(e.g., maximizing short-term versus long-term impacts) –Improve forecasts and influence on procurement

Wrap We should not let short-term, and sometimes controversial applications of NTG, overly influence research and evaluation planning objectives; which should be driven by both short- and long-term perspectives on measuring changes in total societal energy efficiency (and energy consumption) in ways that are robust in the face of changing program strategies and policy regimes…Nor should we let measurement challenges lead to measurement avoidance.