 How to infer causation: 8 strategies?  How to put them together? S519

 Adding value to descriptive data to make our evaluation explicit  Our goal  Using quantitative value to evaluate the quality or value of the evaluand in a particular context.  Build up our conclusions based on a level of certainty  What are values:  „good“, „valuable“, „worthwhile“ S519

 Adding „values“ to descriptive data collected about  Process, outcomes, costs, comparisons, exportabilities; or  Situated dimensions or components  Weighting all the strengthens or weaknesses of these values to draw overall conclusion about the evaluand. How  Importance weighting  Merit determination  synthesis methodology S519

 Before we go for methodology, we have to answer the question:  Whether our data are subjective S519

 Subjective 1: Inappropriate application of personal or cultural preferences/biases:  arbitrary, idiosyncratic, unreliable, highly personal (i.e., based purely on personal preference, cultural biases, gender biases)  Subjective 2: assessment or interpretation by a person, rather than guidelines  Using well-founded expert judgments  Robust evidence  Subjective 3: about a person‘s inner life or experiences (e.g., headaches, fears, beliefs, emotions, stress)  Usually not independently verifiable S519

 We provide our conclusion based on certainty in the relevant decision-making context  Keep the whole evaluation well documented and justified  All evaluations, especially high-stakes ones, should be meta-evaluated (i.e., evaluation itself should be evaluated) S519

 Importance determination is the process of assigning labels to dimensions or components to indicate their importance.  Importance weighting  Prioritize improvements  Identify whether identified strengths or weakness are serious or minor  Work out whether an evaluand with mixed results is doing fairly well, quite poorly, or somewhere in between. S519

 Different evaluations  Dimensional evaluation  Looking at multiple dimensions of merit that pertain to the evaluand as a whole rather than separately to its parts.  Component evaluation  Looking at each of the evaluand‘s components (or parts) separately and then synthesizing these findings to draw conclusion about the evaluand as a whole.  Each component can be evaluated on several dimensions that pertain to this component only rather than to the evaluand as a whole.  Holistic evaluation  Looking evaluation as a whole without division into dimensions or components S519

 Component analysis  Evaluating policies, programs, or interventions that have several quite distinct parts  An international program consisting of projects implemented in different locations (e.g. „WIC“ in IU)  A government policy includes multiple policy measurements (e.g. Juvenile delinquency)  An organizational transformation includes several distinct interventions (e.g. Career support) S519

 Dimensional evaluation  Entities whose quality or value is experienced by consumers on multiple dimensions that pertain to the evaluand as a whole  Product evaluation (i.e. Car evaluation) S519

 Holistic evaluation  Unusual in the evaluation of programs, policies and other large complex evaluands.  More common in personnel, product and service evaluation (expertise-oriented evaluation)  Judging the overall quality of a sample of writing  Grading essays  Classroom teaching  Athletic performance  cosmetics S519

 Weak performance on minor criteria (e.g. dimensions, components) may be no big deal,  But weak performance on important criteria can be very serious issues. S519

 1. having stakeholders or consumers „vote“ on importance  Commonly used in both participatory and nonparticipatory evaluations  Collecting opinions from everybody  Assumptions  Each person is well informed  Stakeholder‘s belief what (s)he chooses is important  Stakeholder‘s important should be treated equally  Pros and cons? S519

 2. Drawing on the knowledge of selected stakeholders  Using selected stakeholder input to guide the assignment of importance weightings  Collecting opinions from selected experts  Setting up the Bars  A bar is a defined minimum level of criterion performance below which the evaluand is considered completely unacceptable, regardless of performance on other criteria. S519

 2. Drawing on the knowledge of selected stakeholders  Assumptions:  The stakeholders should be sufficiently well informed to provide valuable relevant information  The combination of stakeholder input will provide sufficient certainty about importance for the given decision-making context  Pros and cons? S519

 3. Using evidence from the literature  Literature review  Evaluations of similar evaluations in similar contexts  Research documenting the key drivers (or strongest predicators) of success or failure with this type of evaluand.  Assumptions  The volume and quality of the available research is sufficient to judge the importance  The context of other research is sufficiently similar to yours and therefore that the findings can be reasonably applied to your setting  Pros and cons? S519

 4. Using specialist judgment  When you have tight timeline, no time for gathering stakeholders and looking for literature  Identify one or two (or two or more) well-known specialists in the domain  Better be supplemented with other evidence  Pros and cons? S519

 5. Using evidence from the needs and values assessments  Determining the importance of criteria (dimensions)  Any frequently mentioned characteristics?  Looking for poor-performing evaluators that cause serious problem  Looking for top-notch evaluators that have dramatic impacts on success S519

 5. Using evidence from the needs and values assessments  Determining the importance of components  Severity of dysfunction addressed (primary consideration)  Scarcity of alternatives: no other options for addressing the need.  Intent to use alternatives: if the evaluand component in question did not exist.  Rubrics to measure (Table7.4, 7.5 and 7.6, 7.7, 7.8 (combined))  Prons and cons? S519

 6. Using program theory and evidence of causal linkages  When criteria or components are linked to needs through a complex logic chain.  Such as „soft“ skills or attributes (e.g., inspirational leadership, self-esteem, stress management, a kind of instrumental needs)  More upstream variables (see Exhibit 7.5)  How to estimate the strengths of the links  Interview  Analyze your previous data ... S519

 Always think whether they are applicable  Choose mulitple of them S519

 Table 7.10 (class dissusion)  Form a group  Discuss which strategies you will choose to determine the importance for the “student services in the school health program” (see Table 7.8)  Discuss which strategies you will choose to determine the importance for your group project S519