S519: Evaluation of Information Systems Analyzing data: Synthesis D-Ch9.

Slides:



Advertisements
Similar presentations
Critical Reading Strategies: Overview of Research Process
Advertisements

Performance Evaluation
Chapter 7 Job Description Performance appraisal Job evaluation
Developing a Hiring System OK, Enough Assessing: Who Do We Hire??!!
Laura Pejsa Goff Pejsa & Associates MESI 2014
S519: Evaluation of Information Systems Analyzing data: value and importance Ch6+7.
 How to infer causation: 8 strategies?  How to put them together? S519.
S519: Evaluation of Information Systems Analyzing data: Merit Ch8.
Performance Appraisal
 Systematic determination of the quality or value of something (Scriven, 1991)  What can we evaluate?  Projects, programs, or organizations  Personnel.
 What are evaluation criteria?  What are step3 and step 4?  What are the step3 and step4 output report? S519.
Introduction to Management Science
Definitions Performance Appraisal
SE 450 Software Processes & Product Metrics Survey Use & Design.
Performance Evaluation
Introduction to Management Science
Exhibit 5.1: Many Ways to Create Internal Structure
S519: Evaluation of Information Systems Analyzing data: Rank Ch9-p171.
S519: Evaluation of Information Systems Understanding Evaluation Ch1+2.
EDU555 CURRICULUM & INSTRUCTION ENCIK MUHAMAD FURKAN MAT SALLEH WEEK 4 CURRICULUM EVALUATION.
Evaluating Physical Activity Intervention Programs Thomas Schmid, PhD Physical Activity and Health Branch CDC Atlanta, Georgia, USA.
Quality of life 8th seminar Jaroslav Biolek. Objective x subjective -What is relation between objective and subjective indicators of quality of life (according.
Copyright © 2016 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Writing the Research Paper
Assessment Statements  The Internal Assessment (IA) Rubric IS the assessment statement.
REC 375—Leadership and Management of Parks and Recreation Services Jim Herstine, Ph.D., CPRP Assistant Professor Parks and Recreation Management UNC Wilmington.
9-1 Copyright © 2010 Pearson Education, Inc. Publishing as Prentice Hall Multicriteria Decision Making Chapter 9.
Multicriteria Decision Making
Product Evaluation the outcome phase. Do the magic bullets work? How do you know when an innovative educational program has “worked”? How do you know.
Analyzing data: Synthesis
Evaluation of Quality of Learning Scenarios and Their Suitability to Particular Learners’ Profiles Assoc. Prof. Dr. Eugenijus Kurilovas, Vilnius University,
Program Evaluation. Program evaluation Methodological techniques of the social sciences social policy public welfare administration.
TEA Science Workshop #3 October 1, 2012 Kim Lott Utah State University.
S519: Evaluation of Information Systems
Chapter 6 : Software Metrics
Building Your Assessment Plan Esther Isabelle Wilder Lehman College.
OBSERVATIONAL METHODS © 2012 The McGraw-Hill Companies, Inc.
Course on Data Analysis and Interpretation P Presented by B. Unmar Sponsored by GGSU PART 2 Date: 5 July
Lesson 8: Effectiveness Macerata, 11 December Alessandro Valenza, Director, t33 srl.
WELNS 670: Wellness Research Design Chapter 5: Planning Your Research Design.
An Introduction to Measurement and Evaluation Emily H. Wughalter, Ed.D. Summer 2010 Department of Kinesiology.
Setting the Stage: Workshop Framing and Crosscutting Issues Simon Hearn, ODI Evaluation Methods for Large-Scale, Complex, Multi- National Global Health.
S519: Evaluation of Information Systems Analyzing data: Causation Ch5.
Chapter 9 - Multicriteria Decision Making 1 Chapter 9 Multicriteria Decision Making Introduction to Management Science 8th Edition by Bernard W. Taylor.
S519: Evaluation of Information Systems Analyzing data: Synthesis D-Ch9.
S519: Evaluation of Information Systems Analyzing data: value and importance Ch6+7.
 Read through problems  Identify problems you think your team has the capacity and interest to solve  Prioritize the problems and indicate the.
Evaluation Workshop Self evaluation – some workable ideas and approaches.
 What is synthesis methodology?, why do we need that?  What is synthesis for grading?  Quantitative  Qualitative  How to merge all the conclusions.
S519: Evaluation of Information Systems Result D-Ch10.
 Now we are ready to write our evaluation report.  Basically we are going to fill our content to the checklist boxes we learned in lec2. S519.
META-ANALYSIS, RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON.
Assessment and Testing
S519: Evaluation of Information Systems Analyzing data: Merit Ch8.
Personnel record and report is a statement describing an event, situation and happening in a clear manner. It provide both qualitative and quantitative.
OBSERVATIONAL METHODS © 2009 The McGraw-Hill Companies, Inc.
Research Methodology For AEP Assoc. Prof. Dr. Nguyen Thi Tuyet Mai HÀ NỘI 12/2015.
Fidelity of Implementation A tool designed to provide descriptions of facets of a coherent whole school literacy initiative. A tool designed to provide.
S519: Evaluation of Information Systems Understanding Evaluation Ch1+2.
Program Evaluation for Nonprofit Professionals Unit 4: Analysis, Reporting and Use.
S519: Evaluation of Information Systems Analyzing data: Rank Ch9-p171.
Research Methodology II Term review. Theoretical framework  What is meant by a theory? It is a set of interrelated constructs, definitions and propositions.
Aspect 1 Defining the problem - Problem: The design context will normally offer a variety of potential problems to solve. A focused problem and need is.
Program Evaluation “Most often defined as a process used to determine whether the design and delivery of a program were effective and whether the proposed.
RES 320 expert Expect Success/res320expertdotcom FOR MORE CLASSES VISIT
Right-sized Evaluation
Academic Rubric Slides
MANA 4328 Dr. Jeanne Michalski
Chapter 7 Job Description Performance appraisal Job evaluation
S519: Evaluation of Information Systems
Presentation transcript:

S519: Evaluation of Information Systems Analyzing data: Synthesis D-Ch9

Last week What are merit determination? What is absolute merit? How to draw rubric? What is relative merit? How to draw rubric?

Synthesis methodology It is a tool to allow us to draw overall evaluative conclusions from multiple findings about a single evaluand. Synthesis is the process of combining a set of ratings or performances on several components or dimensions into an overall rating.

Synthesis methodology KEC Box 11: Overall Significance What are the main areas where the evaluand is doing well Is this the most cost-effective use of the available resources to address the identified needs without excessive adverse impact?

Synthesis methodology Merit determination To develop the rubrics To use rubrics to summarize the multiple findings A kind of synthesis method Rubrics are one of the simplest methods to blend data. But when data is a bit more complex, it is difficult to use a rubric as the only tool Data are not equally important or reliable Different nuances and combinations (such as Table8.3)

It is not It is not meta-analysis A special statistical techniques to give a weighted average of effect sizes across multiple studies – for quantitative studies It is not literature review A judgment from a reviewer’s point of view.

It is A systematic way of taking into account the pluses and minuses uncovered when the evaluation team draws the evaluative conclusions. Some synthesis is always necessary, whether the evaluation is formative or summative It combines the data we collected and uses comparison to place the evaluand’s performance in a wider context.

Keep in mind Doing poorly on some minimal important criteria Doing poorly on some crucial criteria Are very different!

Costs and comparisons in synthesis Evaluation is Not just: Did the value of the outcomes outweight the value of the resource it took to achieve them? But: Is this evaluand the best possible use of available resources to achieve outcomes of the greatest possible value?

Evaluation Sythesis for „ranking“ If it is „ranking“ (relative) evaluation: Consider each alternative and make explicit comparisons Synthesis for „grading“ If it is „grading“ (absolute) evaluation: Consider different context settings and provide better interpretation of merit

Qualitative or quantitative Quantitative synthesis Using numerical weights Qualitative synthesis Using qualitative labels

Synthesis for “grading” The primary evaluation question is for absolut quality or value How well did the evaluand perform on this dimension? How effective, valuable, or meritorious is the evaluand overall? Is this component worth the resources put into it?

Quantitative weighting example with „bars“ Case: Personnel evaluation in a small accounting firm 13 defined tasks (e.g., telephone, reception, data entry, etc.) Each employee has responsibiltiy for 4-6 tasks Evaluation: Importance weighting (through the voting of the selected stakeholders) In-depth discussion with business owners Derive the importance metric and bars

Quantitative weighting example with „bars“ Evaluation Define the levels of importance: 3 to 5 levels work well in most case Do not go to too many levels (why? Is this useful?) For example task  1. minor task (1)  2. normal-priority task (2)  3. high-priority task (3)  4. extremely high-priority task (4)

Quantitative weighting example with „bars“ Evaluation Setting up rubrics for each 13 tasks Normally 4-6 level is sufficient Example: Performance Rubric  1. Totally unacceptable performance (1)  2. Medicore (substandard) performance (2)  3. Good performance (expected level) (3)  4. Performance that exceeded expectations (4)  5. All-around excellent performance (5) Synthesis – draw the overall conclusion See Exhibit 9.2 (p158)

Exercise Peronal evaluation in a small accouting firm TasksImportanceScore for Alice Telephone12 Data entry23 Tax data management 41 Client support45 Reporting33 Communicating13 How about Alice according to Exhibit 9.2? Lab

Exerice How about Alice Mean = 1*2+2*3+4*1+4*5+3*3+1*3/( ) =44/15 =2.93 She has the good performance?

Qualitative weighting example 1 (with no „bars“) Case: a school-based health program evaluation It contains 9 different components: nutrition education, mental health services, safer sex, legal service and others. How to evaluate these system in low-budget and short period of time whether they are meeting important needs of the students and their families Evaluation: Interview Student surveys

School health system evaluation Survey question design: Two quantitative questions How useful was the program to you? (4-point response scale: not at all useful, somewhat useful, useful, very useful) How satisfied were you with the program? One qualitative question (open-end)? What other changes or events, good or bad, have happened to you or someone you know because of receiving the service?

School health system evaluation Survey result about nutrition system shows in Table 9.1 Look at table 9.1, think about: How can you draw a conclusion from this result about the nutrition system? Is it good or bad? Pay attention of N in Table 9.1

School health system evaluation Setting the importance for these three questions (1-strongest data, 3=weakest data) 1. Ratings of usefulness (directly related to needs) 2. Responses to the open-ended question 3. Satisfaction ratings Creating rubrics for each question Table 9.2 for question 1 and question 2 Table 9.3 for open-ended question

School health system evaluation How to grade the nutrition system based on the first two quantitative questions: Based on Table 9.1, come out with the rubric as Table 9.2 Why 90% is select, 70%-90%.. how to draw Table 9.2 from Table 9.1 and collected data?

School health system evaluation Table 9.3 Rubric for converting data from qualitative evaluation - open-ended responses into merit ratings Is that a good way to do this? Are you happy with this table? If not, how do you want to improve it?

School health system evaluation Synthesis to draw overall conclusion Step-by-step Start with the strongest data (question 1) Blend with open-ended comments Finally take the satisfaction ratings into account See table 9.4 for the whole process

School health system evaluation How to draw final conclusion? Usefulness ratings Satisfaction ratings Open-ended comments Final coclusion: Merit of the nutrition program See table 9.4 Discuss how to apply this to your group project Using quantitative ratings to draw the suggested results and using qualitative ratings to find the positive or negative facts to re-adjust the results

Exercise Form your group project Discuss this case (p ) How to apply this to your group project? Lab

Qualitative (nonnumerical) weighting example 2 Bar A minimum level of performance on a specific dimension Performance below this cannot be compensated for by much better performance on other dimensions (see Exhibit 9.2) Hard hurdle (also referred as global bars) Overall passing requirement for an evaluand as a whole (see Exhibit 9.2) Soft hurdle Overall requirement for entry into a high rating category Place a limit on the maximum rating (e.g., I want all As for my classes)

Qualitative (nonnumerical) weighting example 2 Case: Evaluation of the learning capacity of a small biotechnology start-up company „biosleep“. Evaluation 27 subdimensions of organizational learning capacity (see table 9.5) Data collection: survey and interview Rubric: similar as Table8.2 Importance is built by using strategy 6 in Chapter 7 Using program theory and evidence of causal linkages (p )

Biosleep Evaluation Synthesis Pack the ratings on the subdimensions into 8 main dimensions Combine the ratings on these 8 main dimensions to draw an overall conclusion

Biosleep Dimension by dimension Layer by layer Sub-dimnention1 Sub-dimnention2 Sub-dimnention3 Sub-dimnention4 Dimnention1 Dimnention2 Overall rating

Biosleep Synthesis Subdimensions  Dimensions Using Table 9.6 to draw conclusions of dimentions based on subdimensions Using Table 9.6 to judge Table 9.5 and come out the result as Exhibit 9.4 Dimensions  overall evaluation Based on Table 9.7 (created based on literature review, What is your conclusion for the evaluation of Biosleep? And why?

Exericse Form your group project Discuss on how are you going to grade your evaluation? Which example you would like to follow? How to develop rubric for dimension and overall? Lab