S519: Evaluation of Information Systems Analyzing data: Rank Ch9-p171

Last week What is synthesis methodology?, why do we need that? What is synthesis for grading? What is the rubric for dimensions What is the rubric for components How to merge all the conclusions to get the final grade?

Synthesizing for „ranking“ What are „ranking“ evaluations? Examples? Difference comparing with „grading“ evaluation?

Qualitative and quantitative Qualitative Qualitative weight and sum (QWS) Quantitative Numerical weight and sum (NWS)

Numerical Weight and Sum (NWS) It is a quantitative synthesis method for summing evaluand performance across multiple criteria. It includes Assign numerical importance weight and a numerical performance score to each criteria (dimention) Mutliply weights by performance scores Sum these products The summing result represents the overall merit of the evaluand

Numerical Weight and Sum (NWS) It fits for There are only a small number of criteria (why) There is some other mechanism for taking bars into account (why) There is defensible needs-based strategy for ascribing weights.

Training program evaluation A comparative evaluation on three different interventions for training managers A mountain retreat featuring interactive sessions with multiple world-class management gurus An in-house training and mentoring program run by human resources, A set of videos and latest book on management from managment guru Peter Drucker

Training program evaluation Needs assessment for this evaluation Bear in mind that this is a comparison evaluation How do you want to compare these programs, what are the key features of the programs Identify the dimension of merit (Process, Outcomes and Cost) Decide the importance of the merit (giving weights to merits, based on needs?) See Table 9.8

Training program evaluation Next steps Data collection (what are your experiences for your project data collection?) Data analysis Rate their performance based on pre-defined ratings: excellent, very good, good, fair, or poor) (see Table 9.9 for this example) Convert weights into numbers (see Table 9.10) Convert ratings into numbers (see Table 9.10) Synthesis step (how? See Table 9.11) How to interpret Table 9.11

Exercise Do it by your own hand: Converting Table9.9 to Table9.10 (defining your own numeric value for importance and grading scales) and try to find out which program is the best comparing with others. If suddently, the cost criterias become extremely important, will this change the final result? Work on your own Form the pair and discussion Pros and cons for NWS?

Qualitative Weight and Sum (QWS) It is non-numerical synthesis methodology for summing the performances of an evaluand on multiple criteria to determine overall merit. It is a ranking method for determining the relative merit of two or more evaluands It is not suitable for grading It fits for Personnel selection, products/service/proposal selection

QWS Step1: Determine importance in terms of maximum possible value How (see Chapter 7, six strategies) Table 9.12 (compare with Table 9.8) Step2: Set bars Bar is the cut point between acceptable and unacceptable criteria. Such as: Too expensive to afford Too long away from their work

QWS Step3: Create value determination rubrics Rubrics are level-based (see Chapter 8) Description on each level, how to deal with bar?  Unacceptable  no noticeable value  marginally valuable  valuable  extremely valuable  Such as what performance would look like at each level Each dimension can have its own rubrics or each group of dimensions can have their own rubrics Each group of questions can have their own rubrics Synthesis step can have its own rubrics Example: Rubric for rating finanical cost of training (see table 9.14) How to set up similar rubric for Table 9.12 on extremely valuable criteria (see Table 9.15)

QWS Step4: Check equivalence of value levels across dimensions The validity of the QWS method is highly dependent on ensuring the rough equivalence on the value levels defined for each dimension For example, whether table 9.14 and table 9.15 have the roughly equivalent value levels How to do that? Put them into a matrix. See table 9.16

QWS Step5: rate value of actual performance on each dimension Rating table 9.9 according to rubric (table9.16) See Table 9.17 Step6: tally the number of ratings at each level and look for a clear winner For each program, how many symbols they got? Throw out programs with unacceptable ratings, see whether there is a clear winner?

QWS Step7: refocus Delete the rows with similar score (see table9.18) Count how many symbols each of them got? Can we find the clear winner? Yes or no? Why? How should we go further?

Exercise Form a group to work on this Library wants to subscribe to one of the three magazines and ask you to conduct an evaluation and propose the best solution: Journal of the American Society for Information Science and Technology Journal of Information Science Scientometrics (or choose some magezines you are familiar with) List the criteria you think are important Following the steps of NWS and QWS and tell me your findings and show me the justification of your findings.