 What is synthesis methodology?, why do we need that?  What is synthesis for grading?  Quantitative  Qualitative  How to merge all the conclusions to get the final grade? S519

 What are „ranking“ evaluations?  Examples?  Difference comparing with „grading“ evaluation? S519

 Qualitative  Qualitative weight and sum (QWS)  Quantitative  Numerical weight and sum (NWS) S519

 It is a quantitative synthesis method for summing evaluand performance across multiple criteria.  It includes  Assign numerical importance weight and a numerical performance score to each criteria (dimension)  Multiply weights by performance scores  Sum these products  The summing result represents the overall merit of the evaluand S519

 It fits for  There are only a small number of criteria  There is some other mechanism for taking bars into account (why)  There is defensible needs-based strategy for ascribing weights. S519

 A comparative evaluation on three different interventions for training managers  A mountain retreat featuring interactive sessions with multiple world- class management gurus  An in-house training and mentoring program run by human resources,  A set of videos and latest book on management from management guru Peter Drucker S519

 Needs assessment for this evaluation  Bear in mind that this is a comparison evaluation  How do you want to compare these programs, what are the key features of the programs  Identify the dimension of merit (Process, Outcomes and Cost)  Decide the importance of the merit (giving weights to merits, based on needs?)  See Table 9.8 S519

 Next steps  Data collection (what are your experiences for your project data collection?)  Data analysis  Rate their performance based on pre-defined ratings: excellent, very good, good, fair, or poor) (see Table 9.9 for this example)  Convert weights into numbers (see Table 9.10)  Convert ratings into numbers (see Table 9.10)  Synthesis step (how? See Table 9.11)  How to interpret Table 9.11 S519

 Do it by your own hand:  Converting Table9.9 to Table9.10 (defining your own numeric value for importance and grading scales) and try to find out which program is the best comparing with others.  If suddenly, the cost criteria become extremely important, will this change the final result?  Work on your own  Form the pair and discussion  Pros and cons for NWS? S519

 It is non-numerical synthesis methodology for summing the performances of an evaluand on multiple criteria to determine overall merit.  It is a ranking method for determining the relative merit of two or more evaluands  It is not suitable for grading  It fits for  Personnel selection, products/service/proposal selection S519

 Step1: Determine importance in terms of maximum possible value  How (see Chapter 7, six strategies)  Table 9.12 (compare with Table 9.8)  Step2: Set bars  Bar is the cut point between acceptable and unacceptable criteria. Such as:  Too expensive to afford  Too long away from their work S519

 Step3: Create value determination rubrics  Rubrics are level-based (see Chapter 8)  Description on each level, how to deal with bar?  Unacceptable  no noticeable value  marginally valuable  valuable  extremely valuable  Such as what performance would look like at each level  Each dimension can have its own rubrics or each group of dimensions can have their own rubrics  Each group of questions can have their own rubrics  Synthesis step can have its own rubrics  Example: Rubric for rating finanical cost of training (see table 9.14) S519

 Step4: Check equivalence of value levels across dimensions  The validity of the QWS method is highly dependent on ensuring the rough equivalence on the value levels defined for each dimension  For example, whether table 9.14 and table 9.15 have the roughly equivalent value levels  How to do that? Put them into a matrix.  See table 9.16 S519

 Step5: rate value of actual performance on each dimension  Rating table 9.9 according to rubric (table9.16)  See Table 9.17  Step6: tally the number of ratings at each level and look for a clear winner  For each program, how many symbols they got?  Throw out programs with unacceptable ratings, see whether there is a clear winner? S519

 Step7: refocus  Delete the rows with similar score (see table9.18)  Count how many symbols each of them got  Can we find the clear winner?  Yes or no?  Why?  How should we go further? S519