Project Workshops Results and Evaluation

General The Results section presents the results to demonstrate the performance of the proposed solution. It shows whether the proposed solution meet our individual objectives. The Evaluation section critically evaluates the proposed solution against the project objectives. It shows how well it meets our objectives. Issues to consider include adequacy, coverage, efficiency, productiveness, effectiveness, elegance, and user- friendliness.

These two sections in effect summarize the project values and the contributions. Examiners pay particular attention to these two sections to find out what you have achieved and your evaluation of the project. However excellent other sections may be, if there are few results (or evaluation), there will be few marks.

The Results Section: Scope The Results section focuses on a smaller picture. It discusses what you have achieved from individual experiments. It is essentially a fact checking.

The Results Section: Content Identify suitable experiments to demonstrate that the proposed solution does (or does not) deliver the individual claimed objectives. The setting of each experiment needs to be stated here. Results are then shown and explained. Results from each experiment are analysed and the implications of them are also presented.

State clearly if the corresponding objective has been met. Some of these analyses and discussions may overlap with the evaluation – find a suitable place to put them. If you have done lots, you may focus on the more interesting points – better to talk advanced stuff than basic deliverables!

The Evaluation Section: Scope The Evaluation section looks at the overall picture. It considers how good the thing is that you have done or created. Evaluation is generally more difficult. You need to think carefully about the questions and how to get answers.

The Evaluation Section: Content There are some questions that you should try to answer: 1.Does your solution solve the problem you set? –A yes-or-no answer.

2.How well (or how worst) does your solution address the problem? –Attempting to measure the effectiveness of the proposed solution. –This is harder, but may yield some insight. 3.Move on to questions of whether and how we can do better if we are to conduct the same project again.

Some Issues Plan ahead what experiments need to be conducted. May reference to relevant literature to see what kind of experiments are typically conducted in your project area. If there are too many experiments to conduct, identify the most suitable ones that are most relevant and most convincing to the objectives. Experiments involving very large scale tests (e.g., large scale user studies) may not be feasible within the limited time of this project. May conduct ‘pilot’ studies instead, but need to indicate their limitations.

Evaluation should be as objective as possible. Need to discuss the limitations of the proposed solution as well as its advantages. This will show that you have thought well about the problem. Evaluation may lead to further experiments to verify the findings. Need to leave time for such possibility.

Usability Studies It is an undergrad favourite! “I asked a few of my friends, right, and they said yeah, looks ok.'' There are some common techniques. You need to identify what particular questions you are trying to answer, then select the appropriate techniques. This may require some deep thinking about what you are trying to do, and what techniques and measures may produce valid information. You may need to justify your choices here.

Questionnaires Often used, but some may be flawed or weak. Good questionnaire design, for use in valid experiments, is not easy. Think very carefully about the point and rationale of questions – e.g., do they really address the question you want? Prevent using suggestive questions. For example: “This interaction method is efficient. How do you rank this method?” Instead, we may simply ask “How do you rank this method?”

More on Evaluation Testing sometimes gets done as an afterthought. There is a risk of doing this for evaluation too. Like testing, evaluation should be taken VERY seriously and even planned for in the early stages of the project! Orient the rest of your work towards something that is interesting and worthwhile to evaluate. It helps avoid doing something vague which ends up being hard to evaluate.

SE Evaluation The Product –a plan will be composed and conducted with regard to the acceptability of the developed system. This may take many forms including quantitative and subjective evaluation. For instance, typical approaches include user acceptance tests, surveys, comparisons to a pre- defined evaluation framework. The Process –this will evaluate the management techniques adopted within the project and provide a critique of their suitability recommending improvements where necessary.

Summary Results are where you clearly present what you have achieved, with evidence. Evaluation is hard, because it asks deeper questions – about how good your solution is. Good evaluation takes time - but do not skimp on it. Doing well here is essential for high marks.