1. S519: Evaluation of Information Systems Evaluation Criteria Ch3+4

2. Last week What is evaluation? What are the steps involved? What are step1 and step2? What are absolute or relative merit?

3. Step3: Defining evaluative criteria Why not goal? Too generic IBM goal: independence in computers (http://query.nytimes.com/gst/fullpage.html?res=9404E7D6 1239F934A15757C0A9679C8B63)http://query.nytimes.com/gst/fullpage.html?res=9404E7D6 1239F934A15757C0A9679C8B63 SAP ERP Goal  SAP provides a comprehensive range of enterprise software applications and business solutions to empower every aspect of your business. … The functionality to optimize your operations – and resources to extend best practices to your entire value chain. D-Ch3-p26-27 Goal-free evaluation (GFE): needs-based evaluation (based on needs rather than goal)

4. Step3: Defining evaluative criteria To build a criterion list, consider the following procedures: A needs assessment Logic model of linking the evaluand to the needs An assesment of other relevant values, such as process, outcomes, and cost A strategy to organize your criterion checklist Make sure that you go into the evaluation with a well-thought-out plan so that you know what you need to know, where to get that information, and how you are going to put it together when you write up your report.

5. Needs assessment Needs that we identify become the outcome criteria we use for the evaluation The data collected during the needs assessment phase can be used as baseline data for comparison if we wish to track change in certain outcome variables.

6. Needs assessment Understand the true needs of your evaluation end users (consumers or impactees) Who are your end users? They are the person or entity who buys or users a product or service, enroll in a training program,etc. Upstream stakeholder (i.e. People on upper level of the structure – manager, designer) Immediate recipients (i.e. People who directly consume your product or service – consumer, trainee) Downstream consumers (i.e. People who indirectly involved in your evaluation)

7. Exercise Quiz: Timing of the needs assessment It is likely that you will run into a few people who will tell you that the only time a needs assessment is appropriate is at the design stage of the program, project, product, or policy. Once the evaluand has been launched, it is way too late for a needs assessment. You need to have a good answer for this one. What should it be? (jot it down on half a page or less, choose the evaluation case from Table 3.2 –D- p32) Form a group to discuss

8. Needs assessment Rules to follow during the evaluation Systematic (step-by-step, thorough) Objective (free from bias) Transparent (easy to repeat and follow)

9. Understanding needs Needs vs. Wants Difference and why A need is something without which unsatisfactory functioning occurs. Different kind of needs Context dependence Conscious needs vs. Unconsious needs Needs we know and needs we do not know Met needs vs. Unmet needs Building a factory (increase job, but create pollution) Performance needs vs. Instrumental needs „need to do“ something for satisfactory functioning (actual problems) vs. Proposed solutions Access email vs. Lightweight laptop Most of the case, performance needs is considered, but not the instrumental needs

10. Needs assessment method Two phases: Identifying and documenting performance needs Investigating the underlying causes of performance needs

11. Identifying performance needs StepsExamples Document the extent of the presenting need by gathering evidence - Ask police for teen drug arrests Find out more about individuals- Is the drug use more prevalent among boys or girls Look for unnoticed performance needs - e.g. By combining of open-ended inquiry -Have there also been problems with violence, or other crimes - asking parents, teachers or community members Come out the clear picture of the needs - Lay out the evidence Identify the performance needs for an increase in drug abuse by teenagers

12. Investigating the cause of the needs Using logic model Training program Improved skills Improved performance If we implement this We will address this underlying need Which should solve our performance problem

13. Exercise If we implement thisWe will address this underlying need Which should solve our performance problem if that was the true underlying need Training programUnskilled employee Education and communication Improved performance Performance incentivesUnmotivated staff Job enrichmentUnchallenged job Provision of resourcesLack of resources Flexible schedulingHome and work conflict Organizational climate intervention Bad working enviornment

14. Exercise Quiz Suppose that you are asked to do a needs assessment for either an after-school program for urban youth or a PDA for high school students. If you asked potential participants or users, „what do you need?“ what kinds of needs would you most likely be tapping into? (circle one type of needs on each line): Consious needs or unconscious needs? Met needs or unmet needs? Performance needs or instrumental needs? List the types of needs that you would still be missing after asking that question

15. More to do Asking potential participants about their needs is just the tip of the iceberg Many important information has to be gathered in some other way Working with experienced upstream impactees Working with project officer for grant writing training program Identifying correct cause by asking open-end inquiry to downstream and upstream impactees  Poor involvement of grant writing  Lack of motivation – increase incentive  Lack of confidence – build up self confidence

16. Other relevant criteria Think about other criteria based on Process evaluation Outcome evaluation Comparative cost-effectiveness Exportability Like: legal, ethical, authenticity, scentific, economic, historical requirements, etc. Being diagnostic Collecting facts

17. Exercise Do it with your project group Study sample example – grantsmanship workshop (D-p42-49) What are the key issues? What do you learn from this example and how to implement this into your evaluation project? Lab

18. Exercise For your evaluation project Outline how you would go about conducting a needs assessment Identify your primary consumers and downstream impactees Indicate how you would identify met and unment needs, consicous and unconsious needs, performance or instrument needs Draw a table like table 3.4 (D-p48), keep left column the same (feel free to add more), in the right column outline the considerations for your project (considering process, outcomes and cost for each criteria if possible) Being critical and diagnostic Lab

19. Step3: output report Needs assessment Identify consumers or impactees (e.g. Table3.2) Identify different needs (e.g. Table3.3) Logic model (e.g. Exhibit3.6 and Exhibit3.7) An assessment of other relavent values with the consideration of process, outcome and cost (e.g. Table3.4) Organizing your criteria see step4 output report

20. Step4: Organizing criteria and indentifying sources of evidence Rolling design unanticipated criteria may add to the list Existing criteria may be modified Never draw a conclusion based on a single piece of evidence Try to view the same issue from different angles through: Different types of data (both qualitative and quantitative) Multiple sources of information (e.g., existing document, observations, input from more than one group of stakeholders) Triangulation: using different ways or data to verify the conclusion Step-by-Step: start the small scope of data collecting, extend it slowly later on.

21. Checkpoints When organizing your criteria, always keep the followings in mind: Process How good are the evaluand‘s content and implementation Outcomes How good are the impacts on immediate recipients and other impactees Comparative Cost-Effectives How costly is it? Excessive, quite high, acceptable or reasonable Exportability How can we extend this to other settings?

22. The process evaluation checkpoint Process evaluation Content What the evaluand consists of, i.e., basic components or design) Implementation How well or efficiently the evaluand was implemented or delivered to the consumers who needed it Other features Any other features that make the program good or bad which are not covered by the first two and are not outcomes or cost-related criteria

23. The process evaluation checkpoint Now feed into these three categories with all the process-relevant ingredients we generated from: the needs assessment and other relevant needs (e.g. Table3.3, Table 3.4, Exhibit3.6) Add additional consideration if necessary Example: from table 4.1 (long list) to table 4.2 (short list) categorySubcategories and criteriaSource of evdience Content evaluation Implementation evlauation Others (if necessary)

24. Exercise Form a group Discuss Table 4.1 How they draw such categories from Chapter 3 How to draw a similar table for your evlauation project Dicuss Table 4.2 Show table 4.1 is shortened How to draw a similar table for your evaluation project Document the rationale behind the inclusion or exclusion of criteria and how they are merged into different categories

25. The outcome evaluation checkpoint What is outcome Things that happen as a result of the program Outcomes can affect anyone listed as consumers How to do Based on logic model in step3 (e.g. Exhibit3.6 and Exhibit3.7) Organize them into subcategories See Table4.3 (D-p60) categorySubcategories and criteriaSource of evdience Knowledge, skill and attitude gain Application of knowledge, skill and attitudes

26. Exercise Form the group Discuss Table 4.3 How can you draw table 4.3 from exhibition3.6 and 3.7 from chapter 3 How to draw the similar table for your evaluation project

27. The comparative cost- effectiveness checkpoint Any evaluation has to take cost into account What are costs? Money Time Effort Space Opportunity costs i.e. If our evaluation use this resources, then other department cannot use them for some time, which generates cost.

28. Cost cube (Scriven, 1991) Type of costs Money, resources, time, opportunities Costs When Preparation, implementation, maintenance, evaluation Costs to whom Participants, community organizations, workshop provider, others D-p61

29. Cost cube table PreparationImplementationMaintenanceEvaluation Money Resources Time Opportunity Costs to Whom Comparison with competitors Find out the most important costs for your evaluation Compare your cost with other competitors

30. Exercise Form a group Fill in the cost cube table for grantsmanship workshop Fill in the cost cube table for your evaluation table

31. The exportability checkpoint What elements of the evaluand (i.e., innovative design or approach) might make it potentially valuable or a significant contribution or advance in another setting Think about examples

32. Exercise Form a group Discuss the exportability for grantsmanship workshop Discuss the exportability for your evaluation project

33. Exercise Suppose you are asked to evaluate a training program designed to help young unemployed people seek and obtain work effectively How could you evaluate? If evaluate resume, how would you go about defining what makes a „good“ resume (half page) List at least three types of evidence you would gather from different sources to evaluate the resume Suppose that a cost-conscious program manager suggests that you are being wasteful and should use just a single indicator instead. How would you repond (half page) Form a group to discuss

34. Step4: Output report Checkpoints for Process (e.g. Table4.1, 4.2) Outcomes (e.g., Table4.3) Comparative Cost-Effectives (e.g., cost cube table) Exportability Short summary of potential areas for exportability