Research Design Week 4 Lecture 1 Thursday, Apr. 1, 2004

ISYS3015 Analytical Methods for IS Professionals Agenda Triangulation What is research design Two broad categories of research methodology The data source Measurement instruments Ethical issues Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Questions about assignment 1 Where to find the research problem? What do I expect to review? How do I write the review? Word limit? Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

ISYS3015 Analytical Methods for IS Professionals Triangulation Triangulation of measures Triangulation of observers Triangulation of theory Triangulation of method Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Examples of triangulation Topic: The amount of violence in popular Hollywood films Measures Observers Theory Method Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

What is research design A detailed blueprint specifying how the research will be carried out Basic format of all research process General criteria for a research project Universality Could be carried out by any competent person Replication Control Measurement Data can be measured Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Quantitative and qualitative orientation Different research problems =>Different research design and method Collection of different types of data Different interpretation of data Two broad categories Quantitative research Used to answer questions about relationships among measured variables with the purpose of explaining, predicting and controlling phenomena Qualitative research Used to answer questions about the complex nature of phenomena, often with the purpose of describing and understanding the phenomena from the participant’s point of view Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

ISYS3015 Analytical Methods for IS Professionals Research Process Quantitative Concepts are in the form of distinct variables Relationships are expressed in hypotheses Measures are systematically created before data collection and are standardized Data are in the form of numbers from precise measurement Procedures are standard, and replication is assumed Qualitative Concepts are in the form of themes, motifs, generalizations, and taxonomies Measures are created in an ad hoc manner and are often specific to the individual setting or researcher Data are in the form of words and images from documents, observations and transcripts Procedures are particular and replication is very rare Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

ISYS3015 Analytical Methods for IS Professionals Types of Analysis Quantitative Findings are typically expressed in terms of relationships that are presented in tables and graphs Qualitative Findings are typically expressed by quoting interviews or relating experiences the researcher has had in the field Which approach should I use? Guidelines from “practical application” on page 112 of textbook Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Validity of the methodology Internal validity The extent to which its design and the data that it yields allow the researcher to draw accurate conclusions about cause and effect and other relationships within the data Concerns Lack of control Reactivity: Hawthorne effect Experimenter expectancy 3 cases from textbook page 104 Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Validity of the methodology (cont) External validity The extent to which the conclusions drawn can be generalized to other context Real-life setting A representative sample Replication In a different context Validity in qualitative research triangulation Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Planning for data collection Four fundamental questions about the data What data are needed? Where are the data located? How will the data be secured? How will the data be interpreted? Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

ISYS3015 Analytical Methods for IS Professionals Data source Secondary data Information collected by others for a purpose other than the project at hand, but may be useful for the project Primary data Original data collected for the purpose of the research problem Observation, questionnaire, interview and tape recording, experiment results Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

ISYS3015 Analytical Methods for IS Professionals Use of secondary data Answering research questions or solving some or all of the research problems Helping in problem formulation Providing benchmarking measures and other findings that can be compared later on with the results of the study at hand Building and testing analytical (mathematical, econometric) models Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

An example of data source From this week’s paper “last-minute bidding and the rules for ending second-price auctions: evidence from eBay and Amazon auction” by Roth A. and Ockenfels A. What types of data were collected in the study? How did the authors use those data for? Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Where to find secondary data Sources may be internal or external Internet sites Central and local government studies and report Australian Bureau of Statistics Australian Stock Exchange Commercial research companies selling data Studies and reports of institutions and departments such as universities, telecommunication departments, marketing and other research institutes Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Measurement instruments Use existing measurement instruments A clock to measure execution time A published personality test to measure a person’s tendency to be either shy or outgoing Develop our own measurement instruments A survey to assess people’s opinion about welfare reform A final exam to test students’ understanding of a certain subjects Determine clearly and definitively the nature of the measurement instruments Provide evidence that the the instruments have a reasonable degree of validity and reliability for the research purpose Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Validity of measurement instrument Face validity A personal judgment that a measure seems to be reasonable. Content validity The extent to which a measure reflects the dimension(s) implied by the concept Always in connection with achievement testing Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Validity of measurement instrument Criterion validity The extent to which the results of a measurement instrument correlate with another, presumably related measure (criterion) Construct validity The extent to which the measure reflects the theoretical concept it is supposed to measure Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

Reliability of measurement instrument Test-retest reliability (stability over time) The extent to which the same instrument yields the same result on two different occasions Inter-rater reliability (representative across subgroups) The extent to which two or more individuals evaluating the same product or performance give identical judgments Internal consistency reliability (equivalence across indicators) The extent to which all the items with in a single instrument yield similar results. Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

ISYS3015 Analytical Methods for IS Professionals Ethical Issues Researcher’s ethical responsibility to respondents: Voluntary participation People not be coerced into participating in research Informed consent Prospective research participants must be fully informed about the procedures and risks involved in research and must give their consent to participate. Do not place pressure on respondents Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

ISYS3015 Analytical Methods for IS Professionals Ethical Issues (cont) Confidentiality and Anonymity Identifying information will not be made available to anyone who is not directly involved in the study. Debrief subjects Where experiments or field studies involve deception, subjects should have the study explained to them after the session Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

ISYS3015 Analytical Methods for IS Professionals Ethical Issue(cont) Researcher’s responsibility to his or her discipline and society Represent research literature fairly Acknowledge all your sources Be aware of potential sources of bias Avoid of falsifying data Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

ISYS3015 Analytical Methods for IS Professionals Summary Triangulation General criteria for a research project Two major types of research design quantitative and qualitative design Two different data sources Find and evaluate measurement instruments Ethical issues Thursday, Apr. 1, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney