Warren Leon Clean Energy States Alliance 1

2 You will be connected to audio using your computer’s microphone and speakers (VoIP) or through your headset. If you are calling into the webinar, please select “Use Telephone” on the webinar’s control panel after joining the Webinar. Make sure to enter your Audio PIN, shown in the control panel where you choose the option to join by telephone. Dial: 1 (805) Access code: Webinar ID: Audio PIN: Shown after joining the meeting You will automatically be muted when you join the webinar. We’ll take questions after all the presentations have been made; you can queue up a question anytime for the organizer to ask at the end of the session by typing it in the Question box of the control panel. We can also unmute you to ask your question if you raise your “hand.” This webinar is being recorded and will be made available after the call on the CESA website at in the Members section of the website.

 No monthly call in August  MOUs for have been mailed. Please contact Maria or Anne with any questions at  We will be sending out a preliminary agenda and pre-registration for the October members meeting in Washington, DC, shortly. Please pre-register ASAP! 3

 Explain what is in the report  Make it easier for you to use the report  Cover some of the recommendations in the report  Answer your questions and/or discuss whatever you are interested in 4

 Considers evaluation from the perspective of the program manager  Seeks to help you identify evaluation activities that will be useful, cost-effective, and well- received by program staff, policymakers, and stakeholders  Considers how to choose among and approach different types of evaluations  Serve as a reference guide: designed to make it easy to find the sections relevant to you 5

 How to prepare for effective evaluation ◦ Setting program goals ◦ Producing a program theory ◦ Setting evaluation goals ◦ Choosing an evaluator  Discussion of five types of evaluation, with recommendations for how to approach each  Framing and presenting evaluations for external audiences 6

 Developing a program theory ◦ From California Evaluation Framework  Sample program logic model ◦ From NYSERDA  Possible evaluation questions ◦ From EERE Guide to Managing General Evaluation Studies  Descriptions of models used in cost-benefit evaluations: IMPLAN, JEDI, REMI  Reference works  Representative evaluation reports 7

My Starting Point 1.Improve the quality and efficiency of programs 2.Put a program into context by helping managers & stake-holders understand what it is accomplish- ing and how it compares to other programs 3.Demonstrate that the agency takes its responsibilities seriously and seeks to maximize the public benefits of public spending 8

 Evaluation doesn’t receive enough attention  Few widely accepted protocols for evaluating renewable energy programs  Difficult to evaluate the results of some programs  Program goals may not be explicit or fully thought out  The decision about what to evaluate is not made by program managers 9

“Not everything that counts can be counted, and not everything that can be counted counts.” Albert Einstein 10

1. Needs and Market Assessments 2. Process Evaluations 3. Outcome Evaluations ◦ Some evaluators use “gross outcomes” 4. Impact Evaluations ◦ Some evaluators use “net outcomes” 5. Cost-Benefit Evaluations 11

 What It Does ◦ Identifies target markets ◦ Identifies barriers to the adoption of renewable energy ◦ Understand a market or audience  Why It Is Used ◦ Help program managers design programs ◦ Establish baselines for measuring future progress  Key Comments ◦ They can lead to better programs ◦ They can create baselines for future evaluations of program outcomes and impacts ◦ Clean energy agencies should do them more frequently  12

 What It Does ◦ Examines program implementation processes and operations ◦ Determines whether the program is well-designed, efficiently managed, and effectively marketed ◦ Assess customer satisfaction  Why It Is Used ◦ Identify ways to improve the program ◦ Understand the views of customers and other stakeholders 13

 They come in many shapes and sizes  Renewable energy agencies should do them more frequently— especially small, focused evaluation reports  Customer satisfaction surveys are especially valuable 14

 What It Does ◦ Determines whether the program is achieving its intended outcomes and objectives  Why It Is Used ◦ Keep program managers and others focused on the program’s goals ◦ Know whether a program is achieving its objectives ◦ Determine whether the program should be modified so that it is better achieving its objectives 15

 It is important to know if a program is achieving its goals  Depending upon the program and its intended outcomes, an outcome evaluation can be relatively straight-forward or very complicated  Consider up front whether the dissemination of research findings can help key stake- holders and will move a program towards its goals 16

1. Energy outcomes 2. Environmental outcomes 3. Electricity system outcomes 4. Economic outcomes  EPA report: Assessing the Multiple Benefits of Clean Energy: A Resource for States  For renewable energy installation programs: easier to quantify their energy & environmental outcomes than their electric system & economic outcomes 17

 Market transformation programs  Business development programs  Research, R&D, and demonstration programs  Education and information programs 18

 What It Does ◦ Determines the share of the outcomes caused by the program rather than other factors ◦ Identifies unintended but valuable benefits of the program  Why It Is Used ◦ Understand what the program is actually causing to happen ◦ Determine whether the program is unnecessarily providing funding to free riders who do not need the program to act  19

 The more complicated the program theory and the more multi-faceted the program’s route to achieving outcomes, the harder it is to determine which outcomes would have occurred without the program ◦ Consider how difficult it will be to determine your program’s impacts and how precise an answer you need  It can be helpful to gather the results from evaluations of similar programs in other states ◦ If the evaluation yields results significantly different than other states, have the evaluator provide an explanation  Choose experienced evaluators  Piggy-back other questions onto a survey 20

 What It Does ◦ Compares the economic and/or other benefits of a program’s impacts to the cost of achieving those impacts.  Why It Is Used ◦ Determine the extent to which the program’s benefits outweigh its costs ◦ Understand whether the program is cost-effective ◦ Decide whether the program should be continued as is, modified, or ended 21

 Proceed cautiously  These evaluations usually involve predictions about the future, introducing considerable uncertainty  Consider indirect costs and benefits  Choose experienced evaluators & understand their methodology  Make sure the presentation of results reveals rather than obscures the assumptions and uncertainties Consider low-cost alternatives Commentary by an economist Comparisons between states 22

1. Think carefully about why you want to undertake the evaluation 2. Choose an experienced evaluator (including ones with pre-existing data & models) 3. Make sure you understand the research methods 4. Have the evaluator justify key assumptions 5. Assess more than one scenario 6. Understand the discount rate 7. Ask the evaluator to examine comparable studies & justify differences 8. Understand exactly what the results show 9. Make them comparative 10. Don’t imply the results are more accurate than they are 23

1. Identify all the audiences for the study 2. Make sure the evaluators understand the audiences 3. Develop an outreach plan 4. Decide whether the evaluators should produce collateral material 5. Don’t present the results with inappropriate precision 6. Decide on responses to the report and what to share 7. Disseminate relevant information to stakeholders 24

Warren Leon Senior Advisor Clean Energy States Alliance 25