1 Types of Evaluation Decide on the Purpose: Formative - improve and inform Summative- identify value/effect

2 Summative Evaluation “A study conducted at the end of a program (or of a phase of the program) to determine the extent to which anticipated outcomes were produced.” Weiss, Evaluation, 1998

3 Summative Evaluation (judgment) Phases/Stages Full Implementation (Process measured against criteria for successful implementation) Outcome ( 0utcomes measured against criteria for success) Program or product must be in a state of completion or at an interim ending point. If it is not, then offer formative evaluation.

4 Classify evaluation as…. Formative: using evaluation to improve or inform the development of a program Summative: using evaluation to judge the merit of a program

5 Types of Evaluation Formative Compares status/performance with developer’s hopes. Purpose is correction/ modification, information for decision making (e.g. finding out unknown or testing assumptions) Summative Establishes performance record )

6 Formative Evaluation is the systematic gathering and analysis of information used to advise and improve the conceptualization, design, production, and/or implementation of a wide array of products and programs. Formative evaluation takes place before a product or program has been finalized and is premised upon the considerations, patterns of usage, reactions, and comments of intended learners, viewers, listeners, and/or end-users. The purpose of the evaluation is to inform pending decisions specific to the project in order to increase the likelihood that desired outcomes will be reached. (Holland, 2003, Oryx Dictionary of Education)

7 What is the Purpose of Formative Evaluation? (Flagg, 1990) The purpose is to delineate, obtain, and provide information for making educational decisions ( p. 4) The consistent goal is to maximize the program’s potential for success (p. 34)

8  Practical- concrete questions to inform specific decisions.  Checks on Developers’ Assumptions  Uncover team’s preconceptions about visitor and their interaction with product Distinguishing Characteristics of Formative Evaluation (Fisch & Bernstein)

9 Types of Evaluation Decide on the Purpose: Formative - improve and inform Summative- identify value/effect What Types of Evaluation are available?

10 What Types of Evaluations are Available? Heuristic Evaluations* Expert Evaluations* Needs Assessments* Formative Evaluation of concepts*/product Pilot study Implementation/Process Assessments Outcome Evaluation

11 Formative Evaluation (informs decisions) Phases: Pre-design/Planning/Front-end Design Phase Production Phase (product) Implementation Program or product is not yet consider to be complete or finished. Focus on interim outcomes.

12 Phases of Formative Evaluation (Flagg, 1990) Planning Phase (Front-End) Earliest Phase Working with the Proposal Feasibility of the project Defines the audience Outlines the content States the goals Identifies the media Where and the when it will be used/ experienced

13 Phases of Formative Evaluation (Flagg, 1990) Planning Phase Earliest Phase Front-end analysis Needs assessment Who needs what content? What do people already know about the subject? What might people expect to see? learn?

14 Front End Evaluations (Testing Assumptions by Dierking and Pollock) Used to assess plans before any practical design work is carried out In museum setting, analysis is visitor based In technology setting, analysis is viewer or end-user based In educational setting, analysis is learner based. ASKS: What understandings, feelings, attitudes, expectations and knowledge does the visitor, learner bring to the project?

15 Front End Evaluations (Testing Assumptions by Dierking and Pollock) Make assumptions explicit ( DIA) Identify the ties to visitors everyday life- “hook” Weigh missions and goals vs interest of staff and resources

16 Phases of Formative Evaluation (Flagg, 1990) Design Phase (no programming or filming has been done yet) Decisions are made regarding: specific content objectives instructional strategies

17 Phases of Formative Evaluation (Flagg, 1990) Design Phase (no programming or filming has been done yet) Working with : Writers’ Notebook (TV) outlines goals, content, audience characteristics and gives examples of treatments Story Boards& Flowcharts (Computer) script with pictures or descriptions of the screens

18 Phases of Formative Evaluation (Flagg, 1990) Production Phase ( the camera crew film and computer programmers write codes) Decisions are made regarding issues that effect: appeal comprehensibility user friendliness pacing icons

19 Phases of Formative Evaluation (Flagg, 1990) Production Phase ( the camera crew film and computer programmers write codes) Working with scripts storyboards, flowcharts, rough cuts, prototypes

20 Phases of Formative Evaluation (Flagg, 1990) Implementation Phase (field testing) Evaluating program in designated context with appropriate target group Often done to help the development of support materials like print materials or to inform designers of changes that should be made in future programs.

21 Formative Evaluation (informs decisions) Phases: Pre-design/Planning/Front-end Design Phase Production Phase (product) Implementation Program or product is not yet consider to be complete or finished. Focus on interim outcomes.

22 Needs Assessments Usually inform planning Very often must consider what’s already available to meet the needs. Can these services be utilized? Need to know something about the extent of the problem. Need to know something about the nature of intended audience

23 Evaluating Programs- Various Stages and Purposes:  Planning Assessment of the need for the program within a community (extent of the need) Assessment of background perceptions, habits, knowledge, skills, expectations of the intended audience ( nature of the need). Need to know your audience/environment prior to launching something new or extending or changing something that exits.

24 Two Models of Needs Assessments: Marketing Model Gather feedback in order to learn about and to adapt to the needs of their client population. PURPOSE: TO EXPAND AND EXTEND REACH INTO THE COMMUNITY AND TO BETTER ADDRESS THE NEEDS OF THE COMMUNTIY

25 Two Models of Needs Assessments: Marketing Model Product portfolio analysis ( what do people like? What brings them in? What are they interested in knowing about? What do we offer them that works? Doesn’t work? Product/market expansion analysis ( which demographics are underrepresented?, which demographic groups should be targeted?) GOAL: BE RESPONSIVE TO THE TARGET AUDIENCE

26 Two Models of Needs Assessments: Decision-Making Model Evaluating the options available to the developer in terms of the needs of the audience. Where is the greatest interest? Appeal? Lack of knowledge etc? Goal: Information will inform a choice.

27 Criteria of success- those measurable indicators that will signify whether the goals or objectives are being achieved. (Goals and Objectives are the specific desired outcomes.)