1. Analyzing data: Synthesis

2. Review What is merit determination?
What is absolute merit? How to draw rubric?
What is relative merit? How to draw rubric?
S519

3. Synthesis methodology
It is a tool to allow us to draw overall evaluative conclusions from multiple findings about a single evaluand.
Synthesis is the process of combining a set of ratings or performances on several components or dimensions into an overall rating.
S519

4. Synthesis methodology
Merit determination
To develop the rubrics
To use rubrics to summarize the multiple findings
Rubrics are one of the simplest methods to blend data.
But when data is a bit more complex, it is difficult to use a rubric as the only tool
Data are not equally important or reliable
Multi dimensions or multi components
Different nuances and combinations (such as Table8.3)
S519

5. It is not It is not meta-analysis
A special statistical technique to give a weighted average of effect sizes across multiple studies – for quantitative studies
It is not literature review or a summary
A judgment from a reviewer’s point of view.
S519

6. Keep in mind Doing poorly on some minimal important criteria
Doing poorly on some crucial criteria
Are very different!
S519

7. Evaluation Synthesis for „ranking“ Synthesis for „grading“
If it is „ranking“ (relative) evaluation:
Consider each alternative and make explicit comparisons
Synthesis for „grading“
If it is „grading“ (absolute) evaluation:
Consider different context settings and provide better interpretation of merit
S519

8. Qualitative or quantitative
Quantitative synthesis
Using numerical weights
Qualitative synthesis
Using qualitative labels
S519

9. Synthesis for “grading”
The primary evaluation question is for absolute quality or value
How well did the evaluand perform on this dimension or component?
How effective, valuable, or meritorious is the evaluand overall?
Is this worth the resources put into it?
S519

10. Quantitative weighting example with „bars“
Case: Personnel evaluation in a small accounting firm
13 defined tasks (e.g., telephone, reception, data entry, etc.)
Each employee has responsibility for 4-6 tasks
Evaluation:
Importance weighting (through the voting of the selected stakeholders)
In-depth discussion with business owners
Derive the importance metric and bars
S519

11. Quantitative weighting example with „bars“
Evaluation
Define the levels of importance:
3 to 5 levels work well in most case
Do not go to too many levels (why? Is this useful?)
For example
task
1. minor task (1)
2. normal-priority task (2)
3. high-priority task (3)
4. extremely high-priority task (4)
S519

12. Quantitative weighting example with „bars“
Evaluation
Setting up rubrics for each 13 tasks
Normally 4-6 level is sufficient
Example: Performance Rubric
1. Totally unacceptable performance (1)
2. Mediocre (substandard) performance (2)
3. Good performance (expected level) (3)
4. Performance that exceeded expectations (4)
5. All-around excellent performance (5)
Synthesis – draw the overall conclusion
See Exhibit 9.2 (p158)
S519

13. Exercise Personal evaluation in a small accouting firm Tasks
Importance
Score for Alice
Telephone 1 2
Data entry 3
Tax data management 4
Client support 5
Reporting
Communicating
How about Alice according to Exhibit 9.2?
S519

14. Exercise Personal evaluation in a small accouting firm Tasks
Importance
Score for John
Telephone 1 2
Data entry 3
Tax data management 4
Client support 5
Reporting
Communicating
How about John according to Exhibit 9.2?
S519

15. Exercise Perosnal evaluation in a small accouting firm Tasks
Importance
Score for Chris
Telephone 1 2
Data entry 3
Tax data management 4
Client support 5
Reporting
Communicating
How about Chris according to Exhibit 9.2?
S519

16. Exerice How about Chris
Mean= 1*2+2*3+4*4+4*5+3*3+1*3/( ) =56/15 =3.73
What is Chris‘ performance?
S519

17. Qualitative weighting example 1 (with no „bars“)
Case: a school-based health program evaluation
It contains 9 different components: nutrition education, mental health services, safer sex, legal service and others.
How to evaluate these systems in low-budget and short period of time whether they are meeting important needs of the students and their families
Evaluation:
Interview
Student surveys
S519

18. School health system evaluation
Survey question design:
Two quantitative questions
How useful was the program to you? (4-point response scale: not at all useful, somewhat useful, useful, very useful)
How satisfied were you with the program?
One qualitative question (open-end)?
What other changes or events, good or bad, have happened to you or someone you know because of receiving the service?
S519

19. School health system evaluation
Survey result about nutrition system shows in Table 9.1
Look at table 9.1, think about:
How can you draw a conclusion from this result about the nutrition system? Is it good or bad?
S519

20. School health system evaluation
Setting the importance for these three questions (1-strongest data, 3=weakest data)
1. Ratings of usefulness (directly related to needs)
2. Responses to the open-ended question
3. Satisfaction ratings
Creating rubrics for each question
Table 9.2 for question 1 and question 2
Table 9.3 for open-ended question
S519

21. School health system evaluation
How to grade the nutrition system based on the first two quantitative questions:
Based on Table 9.1, come out with the rubric as Table 9.2
Why 90% is select, 70%-90%..
How to draw Table 9.2 from Table 9.1 and collected data?
S519

22. School health system evaluation
Table 9.3
Rubric for converting data from qualitative evaluation - open-ended responses into merit ratings
Is that a good way to do this?
Are you happy with this table?
If not, how do you want to improve it?
S519

23. School health system evaluation
Synthesis to draw overall conclusion
Step-by-step
Start with the strongest data (question 1)
Blend with open-ended comments
Finally take the satisfaction ratings into account
See table 9.4 for the whole process
S519

24. School health system evaluation
How to draw final conclusion?
Usefulness ratings
Final coclusion:
Merit of the nutrition program
Satisfaction ratings
Open-ended comments
Using quantitative ratings to draw the suggested results and using qualitative ratings to find the positive or negative facts to re-adjust the results
See table 9.4
Discuss how to apply this to your group project
S519

25. Qualitative (nonnumerical) weighting example 2
Bar
A minimum level of performance on a specific dimension
Performance below this cannot be compensated for by much better performance on other dimensions (see Exhibit 9.2)
Hard hurdle (also referred as global bars)
Overall passing requirement for an evaluand as a whole (see Exhibit 9.2)
Soft hurdle
Overall requirement for entry into a high rating category
Place a limit on the maximum rating (e.g., I want all As for my classes)
S519

26. Qualitative (nonnumerical) weighting example 2
Case: Evaluation of the learning capacity of a small biotechnology start-up company „biosleep“.
Evaluation
27 subdimensions of organizational learning capacity (see table 9.5)
Data collection: survey and interview
Rubric: similar as Table 8.2
Importance is built by using strategy 6 in Chapter 7
Using program theory and evidence of causal linkages (p )
S519

27. Biosleep Evaluation Synthesis
Pack the ratings on the subdimensions into 8 main dimensions
Combine the ratings on these 8 main dimensions to draw an overall conclusion
S519

28. Biosleep Dimension by dimension Layer by layer Sub-dimnention1
Overall rating
Sub-dimnention3
Dimnention2
Sub-dimnention4
S519

29. Biosleep Synthesis Subdimensions  Dimensions
Using Table 9.6 to draw conclusions of dimentions based on subdimensions
Using Table 9.6 to judge Table 9.5 and come out the result as Exhibit 9.4
Dimensions  overall evaluation
Based on Table 9.7 (created based on literature review,
What is your conclusion for the evaluation of Biosleep? And why?
S519

30. Exericse Form your group project
Discuss on how are you going to grade your evaluation?
Which example you would like to follow?
How to develop rubric for dimension and overall?
S519