Chapter 11: Experiments Introduction 11.1 Experiment goal 11.2 Reaching the goal 11.3 Experiment design 11.3.1 Experimental and control groups 11.3.2 Single group 11.3.3 Blind experiments 11.4 Conclusion

Introduction Do an experiment to confirm some theory or to find out what would happen if … Want to make generalisations: Conclusions should not be limited to specific cases tested.

11.1 Experiment goal To see if you can find something interesting Play around with data and see what is there to learn. To test a theory Conduct a limited experiment to see if the theory holds for some specific cases To prove a theory To disprove it is only necessary to find a single example for which the theory does not holds. To prove, need: Control over all factors; Representative sampling

11.2 Reaching the goal See fig. 11.1, p 70. Five examples

11.3 Experiment design Split participants randomly into two groups (e.g. placebo in medical research) Participants should not know in which group they fall Participants should not learn from experience, i.e. the same participant may not test system A and thereafter system B.

11.3.1 Experimental and control groups Two groups are treated in exactly the same way. Observe effects & process data Fundamental assumption: Two groups are identical in all other respects. E.g. difference in computer literacy and/or previous computer experience

11.3.2 Single group The same group used first for control experiment and then for experiment itself. No question that the groups may differ, but the control case may provide experience. Especially fruitful if you test hardware and and/or software and not humans.

11.3.3 Blind experiments E.g. placebo in medical experiments Not as easy in IT research Double-blind refers to case where even the observer does not know which group has been exposed to which treatment Hardly ever conducted in IT research

11.4 Conclusion Major concerns Observations are caused by experimental inputs and nothing else (internal validity) Results observed can be generalised (external validity) Compile a protocol (what, when, how, why, where) activities are to be performed