1. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experimental Research Outline: Strategy of experimental research (Platt) What are experiments and how to do experimental research, Different kinds of experimental designs.

2. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experimental Research Strategy (Platt) Systematic and precise method of scientific thinking Accumulative method of inductive inference Can contribute to rapid scientific progress

3. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney

4. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Steps in Experimental Research 1.Devise alternative hypotheses ( Existing theory ) 2.Devise crucial experiments with alternative possible outcomes, each of which exclude one or more possible hypotheses, ( Experiment ) 3.Conduct the experiment, get a clean result, ( Outcome ) 4.Back to step 1, making sub-hypotheses, or sequential hypotheses to refine the possibilities, exclusion and induction ( exclusion and building the inductive (logical) tree)

5. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Inductive Inference Useful for exploring the unknown Distinct from deductive reasoning; nature provides us the answers Challenge is to pose the right questions; clever choice of hypothesis, experiment, outcome, and exclusion.

6. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Origins of the experimental approach Francis Bacon (interconnecting theory and experiment) building the conditional inductive tree, consecutive inductive inferences) Karl Popper, falsificationism, there is no such thing as proof in science, science advances through a series of disproofs or falsification. Assertions in science have to be falsifiable, “.. it must be possible for all empirical scientific systems to refuted by experience”. Fisher’ s work in the 1930’s and 40’s in the area of statistical inference.

7. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experimental Research Methods What is an experiment? Manipulation of one or more variables by the experimenter to determine the effect of this manipulation on another variable. Carefully designed and executed plan for data collection and analysis to test specific hypotheses. Examples of hypotheses? Well-designed experiments can permit causal inferences to be made.

8. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experiment : Features Typically held at the discretion of the researcher. Ability to use various controls to isolate sources of variation. Ability to explore cause-effect relationships.

9. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Phases in Experimental Studies 1.Formulation of one or more hypotheses. -Usually deductions or derivations from theoretical explanations (of the behavioural phenomenon) or strong hunches/speculations. 2.Translation of the hypotheses into a set of treatment conditions and appropriate experimental design. 3.Conduct the experiment, collect the data 4.Statistical analysis of the data, interpretation of the results and writing up.

10. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experimental Design Independent Variables: The variable(s) manipulated in the experiment. (also called manipulated variable, treatment variable or factor). Typically nominal (categorical) variable. Dependent Variable(s) Measure(s) that capture (performance) of the phenomenon

11. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Control or Nuisance Variables Undesired sources of variation in an experiment that affect the dependent variable measurement, Typically of three types: -organismic -environmental -experimental task.

12. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Approaches to control the nuisance variable Random assignment of subjects to treatment groups, Holding the (pre-identified) nuisance variable constant for all subjects, Statistical control using Analysis of Covariance (ANCOVA).

13. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Sources of Bias Experimenter cues Demand characteristics Evaluation apprehension Hawthorne Effect Negativistic subject

14. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experiments - Advantages Possibility of a variety of manipulative and statistical controls, Random assignment of subjects – greater precision and higher confidence in specifying and testing causal relationships, Manipulation Checks possible. May help identify issues and problems previously unrecognised.

15. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experiments - Disadvantages Problems associated with lab settings, Some phenomenon cannot be studied under controlled conditions, Limitations imposed by moral concerns.

16. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Lab experiments High degree of precision and internal validity possible; internal validity is a measure of the degree to which the study establishes that the treatment actually produced the effect – causality. Potentially low external validity – questions about the extent to which the results are generalisable to other populations, settings, organisations, groups, times etc.

17. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Procedures 1.Getting Ethics Committee Approval 2.Cover Story – description and purpose 3.Recruiting participants -Sample selection -Reference to criterion population -Remuneration and motivation 3.Training the participants 4.Preparing the setting 5.Controlled manipulation of independent variable(s) 6.Manipulation checks 7.Precise measurement of dependent variable(s).

18. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Quasi-experimental Designs Used when it is impossible or difficult to perform true, controlled experiments, Particularly in organisational settings Essentially compromise designs Used when time,cost, and practicality are critical. 1.One shot design 2.One group pretest-posttest design 3.Static Group Design

19. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Observational Methods

20. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Survey Methods Typically questionnaire surveys Strengths: Quantitative data Transparency and some level of objectivity Succinct and easily understood Comparability/reproducibility Can deal with complex problems/issues

21. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Survey Methods Surveys: Interviewer completion Respondent completion Types of surveys 1.Household 2.Telephone 3.Mail 4.Customer 5.Captive Group 6.Organisation 7.Web-based

22. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Survey Methods Issues: Response rates Biases and errors

23. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Survey Methods Types of Questions Open ended and pre-coded Measurement of Attitudes and opinions: Likert Scales Attitude Statements Semantic Differential

24. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Survey Methods Questionnaire Construction Validity/Reliability issues Pilot Testing Sample selection Coding the data Statistical Analysis.

25. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experimentation as a research method Controlled Laboratory Experiments or true experiments Non-experimental designs Simulations

26. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Non-experimental studies Observational Studies Case Studies Surveys Correlational Studies

27. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Types of Experimental Designs Completely randomised designs. CR-p Where p is the number of treatment levels of a single independent variable. Randomised block designs. RB-p Subjects relatively homogeneous w.r.t to the same nuisance variable are assigned to the same block. Completely Randomised Factorial Designs: CRF-pq Two or more treatments can be evaluated simultaneously.

28. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney A Conceptual Model for Studying the Effectiveness of DSS

29. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Hypotheses pertaining to main effects of what-if analysis The first hypothesis proposes that the use of what-if analysis would result in better decision performance. As discussed in the Background section, what-if analysis helps to bring structure to a less-structured problem by incorporating the essential elements of the problem into some underlying models. Decision makers can thus evaluate different decision alternatives in a more systematic way. What-if analysis allows decision makers to explore the effects of changes of decision variables on the decision outcome which helps decision makers to gain deeper understanding of the problem. By receiving simulated outcome feedbacks from what-if analysis, decision makers can make necessary adjustments to their previous judgments about the problem, fine-tune their decisions until a best available solution is reached. On the other hand, decision makers without what-if analysis can only evaluate decision alternatives in a relatively intuitive way without any feedback of any sort. As a result, we postulate that what-if analysis should lead to a better decision. H1 :Decision makers using what-if analysis will show a better decision performance than decision makers without using what-if analysis. Decision maDecision makers with what-if analysis will be have greater confidence in their decisions than decision makers without what-if analysis in making decisions.

30. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney User Interface Main Effects Decision makers will perform better when using the interface with graphical input and graphical presentation (GI, GP) as compared to those using the interface with numeric input and graphical presentation (NI, GP) or numeric input and tabular presentation (NI, TP).

31. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Interaction Effect There will be a significant interaction effect between the two main factors on production cost.

32. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experimental Design User Interface Type What-if Factor Numeric Input with Tabular Presentation Numeric Input with Graphical Presentation Graphical Input with Graphical Presentation Without What-if Analysis18 subjects17 subjects With What-if Analysis18 subjects17 subjects19 subjects

33. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Experimental Task Production Planning task Computer-based decision aids developed to simulate the what-if analysis functionality and the different user interface types according to the six treatment groups.

34. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Numeric Input and tabular Presentation

35. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney

36. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney Results User Interface Type What-if Factor Numeric Input with Tabular Presentation Numeric Input with Graphical Presentation Graphical Input with Graphical Presentation Marginal Mean Without What-if Analysis49778.06 (sd=17204.78) 41477.42 (sd=13222.36) 36634.29 (sd=8108.97) 42975.79 With What-if Analysis34125.77 (sd=5695.54) 40572.80 (sd=14399.84) 31917.42 (sd=2457.53) 35378.38 Marginal Mean 41951.92 40981.34 34073.70 38953.63 Table 2 Mean Production Costs (with standard deviations) of the Six Treatment Groups

37. School of Information Technologies Faculty of Science, College of Sciences and Technology The University of Sydney