Boston Geneva San Francisco Seattle Beginning the Evaluation Journey with FSG KCIC Boot Camp March 24, 2010 Prepared for:

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 2 Today’s Discussion Today’s Agenda Things You Need to Do to Evaluate Your Project’s Progress and Outcomes  Articulate your Theory of Change and develop a project Logic Model  Develop an evaluation plan  Execute your evaluation plan  Participate in Knight’s evaluation process

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 3 Evaluation Overview The KCIC Evaluation Has Three Overarching Objectives Knight’s 2 Goals: –Foundations are engaged in helping meet community information needs –Communities are more informed and engaged Evaluate Progress Inform and Improve Program Strategy Promote Learning Inform the Knight Foundation’s program strategy for the Community Information Challenge and identifying opportunities for refinements over time Create data that promotes shared learning within the KCIC grantee community and the field of place-based foundations to increase understanding of how community information needs can be addressed effectively

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 4 What is Evaluation?  The systematic collection of information based on questions of critical importance to the organization or community: What do you want to know? Why do you want to know it? Where will you find what you need to know? What will it look like? How will you use what you learn? Who are the intended users of the information?  To make judgments, inform decisions, and take action  Is an ongoing process; should not be an “event”  Is best when implemented as a collaborative learning activity that builds an organization’s capacity Definitions of Evaluation

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 5 Process The Evaluation Process: Planned, Purposeful, Systematic Develop Recommenda- tions and Action Plans Articulate Theory of Change & Develop Program Logic Model Focus the Evaluation Collect Data Analyze Data and Interpret Findings Determine Design & Data Collection Methods (numbers, words, pictures) Design Implementation Refine Strategy Based on Evaluation Findings Communicating and Reporting

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 6 Instrumental use - findings reflects changes that can be observed. Conceptual use - findings may lead to stakeholders having different kinds of conversations, greater insights into future decisions, and/or greater commitment to the program or initiative. Political or Symbolic use - legitimate uses of the evaluation findings may be applied to lobbying for the program or initiative, securing new or additional funding, or communicating that the evaluation has taken place. Uses of Evaluation How Can We Use Evaluation? Well designed evaluations will serve multiple purposes

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 7 Internal Stakeholders Board members/Trustees Senior Leaders/Executives Program staff External Stakeholders Legislators/Policymakers Project partners The general public The media Researchers Other funders Journalism programs Stakeholders Different Stakeholders Will Have Different Uses for Evaluation Evaluation Stakeholders Identifying stakeholders has implications for: which and how many questions are asked what indicators will be of interest how findings will be communicated and reported how results will be used

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 8 Example This Theory of Change illustrates at a basic level why SVCF is doing what it’s doing The community needs a regional plan to address climate change and sustainability There is a low awareness and understanding among residents of the intricacies and implications of regional development decisions Residents are not engaged in the planning process today Creation of SVCF’s Venture Fund (with investors) Development of a web- based information platform for citizens to learn about potential scenarios Planned convenings on land use and sustainable cities Awareness campaign (KQED) Interactive kiosks Citizens are more informed and aware of key issues Citizens are more engaged in planning processes Regional plan is developed with community input What we are doing to address the issue – how we plan to solve the problem or create the change (Activities) What the issue is and why we are taking action (Assumptions) The change we hope to see if we are successful (Outcomes) Source: FSG interview; used with grantee’s permission Envision Bay Area – Theory of Change

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 9 As You Think About Your Own KCIC Project Outcomes, Consider the Possibilities Outcomes are…the SO WHAT? The short and/or long term changes, results, and impacts from implementing a project, program, or initiative. Changes may be: positive or negative, singular or multiple in knowledge, skills, and/or attitudes in organizational policies, practices, capacity at community level in terms of behavior, values, and attitudes Outcomes What will be different in your community if your project is successful?

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 10 Types of Outcomes  Short term Outcomes Achieved during program timeframe Within program control “Expect to see”  For example: New knowledge Changed opinion/values Increased skills Changed motivation/intent Changed attitudes Changed aspirations New relationships/networks Short term Outcomes Adapted from: Innovation Network, Inc. (2005)

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 11 Types of Outcomes  Intermediate Outcomes Achieved at the end/beyond program timeframe Follow shorter-term outcomes “Want to see”  For example: Modified behavior Changed policies Changed practices Changed social action Changed decisions Modified structures Adapted from: Innovation Network, Inc. (2005) Intermediate Outcomes

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 12  Longer-term Outcomes Achieved after program timeframe Outside direct program control “Hope to see”  For example: Changed human condition Changed civic condition Changed economic condition Changed environmental condition Longer-term Outcomes Types of Outcomes Source: Innovation Network, Inc. (2005)

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 13 Start Building Your Evaluation Plan by Developing Your Own Theory of Change and Identifying Your Project Outcomes 1.From our interviews with each of you, we have attempted to visually depict your project’s theory of change. Review our version of your ToC and change, add/or revise as needed. 2.If your project were hugely successful in three years, what would your target audience be doing, thinking? (long term outcomes) 3.After one year, what would your target audience be doing, thinking? (short term outcomes) 4.What are some indicators that your project is achieving what you set out to do? (outputs) In Practice See online assignment form:

KCIC Boot Camp – FSG Evaluation Presentation (March 2010) © FSG Social Impact Advisors 14 Next Steps  Developing a Project Logic Model and Key Evaluation Questions (Webinar #1)  Determining Indicators of Online and Offline Behavior (Webinar #1)  Choosing an Evaluation Design and Data Collection Methods (Webinar #2)  Analyzing Quantitative and Qualitative Data (Webinar #2)  Strategies for Communicating and Reporting Evaluation Findings to Multiple Audiences (Webinar #3)  Developing an Evaluation Budget (Webinar #3)  Tips for Hiring an Evaluator (Webinar #3) Next Steps in Designing Your Project’s Evaluation Plan Throughout 2010 – FSG will provide up to 2 hours of additional evaluation-related technical assistance to each grantee