 Systematic determination of the quality or value of something (Scriven, 1991)  What can we evaluate?  Projects, programs, or organizations  Personnel or performance  Policies or strategies  Products or services  Proposals, contract bids, or job application  Almost everything in our daily life because before you make decision, you do the evaluation first. Comparison is a kind of evaluation. Scriven, M. (1991). Evaluation thesaurus (4 th ed.). Newbury Park, CA: Sage

 Evaluand  That which is being evaluated (e.g. Program, policy, project, product, service, organization, almost everything)  In personnel evaluation the term is evaluee S519

 Evaluation is for  Find areas for improvement  Generate an assessment of overall quality  Answer the question of „Merit“ or „Worth“ (Scriven, 1991)  Merit is the „intrinsic“ value of something = „quality“  Worth is the value of something to an individual, an organization, an institution – contextualized merit -- = „value“ S519

 Accountability evaluation  It is important to conduct an independent evaluation  i.e. Nobody on the evaluation team should have a significant vested interest in whether the results are good or bad S519

 Step1: understanding the basic about evaluation (ch1)  Step2: defining the main purposes of the evaluation and the „big picture“ questions that need answers (ch2)  Step3: Identifying the evaluative criteria (ch3)  Step4: Organizing the list of criteria and choosing sources of evidence (collecting data) (ch4) S519

 Step5: analyzing data  dealing with the causation/correlation issue (which cause what, why), to avoid „subjectivity“ (ch5+6)  importance weighting: weight the results (ch7)  Meric determination: how well your evaluand has done on the criteria (good? Unacceptable?) (ch8)  Synthesis methodology: systematic methods for condensing evaluation findings (ch9)  Staticistical analysis: Salkind (2007) S519

 Step6: result  Putting it all together: fitting the pieces into the KEC framework (ch10)  Step7: feedback  Meta-evaluation: how to figure out whether your evlauation is any good (ch11) S519

I. Executive Summary II. Preface III. Methodology 1. Background & Context 2. Descriptions & Definitions 3. Consumers 4. Resources 5. Values 6. Process Evaluation 7. Outcome Evaluation 8 & 9. Comparative Cost-Effectiveness 10. Exportability 11. Overall Significance 12. Recommendations & Explanations 13. Responsibilities 14. Reporting & Follow-up 15. Meta-evaluation S519

 Identify the evaluand  Background and context of evaluand  Why did this program or product come into existence in the first place?  Descriptions and definitions  Describe the evaluand in enough detail so that virtually anyone can understand what it is and what it does  How: collect background information, pay a firsthand visit or literature review S519

 Some tips before you start  Make sure that your evaluand is not difficult to access (geolocation, inanimate objects)  Make your evaluand a clearly defined group (avoid abstract and complex system)  Avoid political ramification (assess your boss pet project, university administration)  To avoid your invovlement in the evaluand (to assess a class which you are teaching, etc.) S519

 Metadata discussion group, Brown Bag discussion group, Twitter, SLIS website, Information Visulization Lab website, Media and Reserve Services in Wells Library, IU CAT, SoE website, Chemistry library website  How about something else related to real-world problems  For social good (  Industry and Government talks at KDD214 (  Maybe this will inspire you with a great project to work on

 Utility  To ensure that an evaluation will serve the practical information needs of intended users  Feasibility  To ensure that an evaluation will be realistic, prudent, diplomatic, and frugal  Propriety  To ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results ( IRB: )  Accuracy  To ensure that an evaluation will reveal and convey technically adequate information about the features S519

 Systematic inquiry  Competence  Integrity/honesty  Respect for people  Responsibilities for general and public welfare S519

 Output: one or two page overview of the evaluand and findings  What is your evaluand  Background and context of your evaluand  Description of your evaluand  Try to be as detail as possible S519

 Who asked for this evaluation and why?  What are the main evaluation questions?  Who are the main audience? S519

 A. what is (are) the main purpose(s) of the evaluation?  To determine the overall quality or value of something (summative evaluation, absolute merit)  i.e. Decision making, funding allocation decision, benchmarking products, using a tool, etc.  To find areas for improvement (formative evaluation, relative merit)  To help a new „thing“ to start  To improve the existing „thing“ S519

 Big picture questions:  B. What is (are) the big picture question(s) for which we need answers?  Absolute merit  Do we want to invest this project?  relative merit  How does this project compare with the other options?  Ranking S519

 Your step2 output report should answer the following questions:  Define the evaluation purpose  Do you need to demonstrate to someone (yourself) the overall quality of something?  Or Do you need to find a file for improvement?  Or do you do both?  Once you answer above questions, figure out what are your big picture questions:  Is your evaluation related to the absolute merit of your evaluand?  Or the relative merit of your evaluand S519