Lecture 5 Data Coding and Experimental Research Methods

2 Overview Research Problems: my comments Working with Data Error-Checking Error-Checking Basic Data Preparation Basic Data PreparationMethods Introduction to Experimental Methods Introduction to Experimental Methods

3 Problems and Justifications Research Question/Problem: What is the effect of obtaining a MIMS degree versus a CS MA degree for getting a job in the tech industry? Different types of degrees teach different skill sets. Jobs in the tech industry will look for these skill sets, which should lead to different outcomes for CS students versus MIMS students. Not a justification by itself– this is part of the context of the argument. By itself, this just sets up an argument that will lead to specific hypotheses. Not a justification by itself– this is part of the context of the argument. By itself, this just sets up an argument that will lead to specific hypotheses.

4 Problems and Justifications (Round 2) Research Question/Problem: What is the effect of obtaining a MIMS degree versus a CS MA degree for getting a job in the tech industry? If we can understand how different degrees (CS versus MIMS) give different job opportunities, it has implications for future enrollment in various academic programs. Furthermore, this research could help us to begin the process of differentiating between various fields of study. Nope, still not a justification. These are implications of the research. Nope, still not a justification. These are implications of the research.

5 Problems and Justifications (Round 3) Research Question/Problem: What is the effect of obtaining a MIMS degree versus a CS MA degree for getting a job in the tech industry? Knowing which graduate degree to obtain is a significant problem for those who want the best jobs in the tech industry. Once a student joins a program, he or she makes a large commitment of time and money. Degree-granting programs often entice potential students to enroll based on the potential job prospects that they will have once they receive a degree. Yet, it is not always clear to prospective students what skill sets are offered by different programs, and which skill sets are being sought by the best employers in the tech industry. This research will increase our understanding of the relationship between degree type (CS versus MIMS) and the related skill sets that ultimately lead to technology job outcomes.

6 Data Preparation Basic Data Setup Unique ID for each case Unique ID for each case Numeric responses whenever possible Numeric responses whenever possible Includes categorical, scale, count, etc Variable names Variable names Variable label Variable label Value label Value label Missing Codes Missing CodesRecoding Simple Simple Intermediate (equations, computations) Intermediate (equations, computations)

7 Error Checking Examples Checking Original Variables for Errors Frequencies Frequencies Descriptives Descriptives Checking and Setting “Missing” codes Recoding and Creating New Variables from Existing Variables Frequencies Frequencies Cross-Tabulations Cross-Tabulations

8 Two Example Datasets: Class Data Set Subset of 1993 General Social Survey

Introduction to Experimental Design and Methods

10 Pro’s and Con’s of Experiments Pro’s Gives researcher tight control over independent factors Gives researcher tight control over independent factors Allows researcher to test key relationships with as few confounding factors as possible Allows researcher to test key relationships with as few confounding factors as possible Allows for direct causal testing Allows for direct causal testingCon’s Usually a smaller N than surveys Usually a smaller N than surveys Sometimes give up large amounts of external validity in favor of construct validity and direct causal analysis Sometimes give up large amounts of external validity in favor of construct validity and direct causal analysis Require a large amount of planning, training, and time– sometimes to test relationship between only 2 factors! Require a large amount of planning, training, and time– sometimes to test relationship between only 2 factors!

11 Active versus Attribute Independent Variables Active independent variable(s): The I.V. is given to the participants, usually for some specified time period. It is often manipulated and controlled by the investigator. The I.V. is given to the participants, usually for some specified time period. It is often manipulated and controlled by the investigator. Attribute independent variable(s): A predictor, but a defining characteristic of individuals. Cannot be manipulated. A predictor, but a defining characteristic of individuals. Cannot be manipulated.

12 True Experiments True experiments protect against both time and group threats to internal validity by randomly assigning subjects to treatment and control groups. The treatment (independent variable) is active. If we cannot randomly assign subjects to different groups, then it is a quasi-experiment. The independent variable is active. If we cannot randomly assign subjects to groups because the groups contain the attribute of interest, and if we give all groups the same treatment, then it is an associational non-experiment. The independent variable is not active.

13 Randomization in Sample and Assignment Random Sample System for choosing participants from a population System for choosing participants from a population Generally, the larger the sampling population the better your generalizability becomes. Generally, the larger the sampling population the better your generalizability becomes. Random Assignment Method for assigning participants randomly to experiment conditions Method for assigning participants randomly to experiment conditions

14 Two Essential Criteria in True Randomized Experimental Design (1) Independent Variables must be manipulated (usually by experimenter, sometimes by context) (2) Participants must be assigned randomly to various conditions or groups

15 Pre-test, experimental manipulation and post-testing Pre-test: allows us to check group equivalence before the intervention X is introduced. Experimental manipulation: An independent variable (X) that the experimenter manipulates. Post-test: allows us to check group equivalence after intervention X has been introduced.

16 Common types of true experiments R O O XO O R X O O R O O XO O (1) (2) XO(3) O(4) Pretest-Posttest Control Post-only Control Solomon 4-group

17 Example: Pen Study Question: Do individuals in Japan and the US make differential choices about ‘unique’ versus ‘less unique’ items when given a choice?

18 Pen Study Independent Variable Cultural difference: Japanese students compared to US students Cultural difference: Japanese students compared to US studentsAssignment Subjects were not randomly assigned because they already fell into one of the two groups. Subjects were not randomly assigned because they already fell into one of the two groups. Dependent Variable: Pen layout (3 of one type, 1 of another) Pen layout (3 of one type, 1 of another) Would they choose the ‘common’ pen or the ‘unique’ one? Would they choose the ‘common’ pen or the ‘unique’ one?

19 Example: Trust-Building Study Question: Do increased risk-taking behaviors over time increase interpersonal trust?

20 Trust-Building Study Independent Variable Experiment Condition (3 conditions): Experiment Condition (3 conditions): Fixed partner on every trial, cannot control amount to entrust to partner Fixed partner on every trial, can control amount to entrust to partner Random partner on every trial, can control amount to entrust to partner Assignment Random assignment of participants to one of the 3 conditions. Random assignment of participants to one of the 3 conditions. Same experiment conducted in Japan and US, and comparisons made between the two studies. Same experiment conducted in Japan and US, and comparisons made between the two studies. Dependent Variable Cooperation rate (i.e., whether they returned the coins to the partner or not) Cooperation rate (i.e., whether they returned the coins to the partner or not)

Validity: Revisited

22 Experiment Construct Validity How do we know that our independent variable is reflecting the intended causal construct and nothing else? Contamination Demand Characteristics Demand Characteristics Anything in the experiment that could guide subjects to expected outcome Experimenter Expectancy Experimenter Expectancy Researcher behavior that guides subjects to expected outcome (self-fulfilling prophecy)

23 Solving Expectancy Effects Naïve experimenter Those conducting study are not aware of theory or hypotheses in the experiment Those conducting study are not aware of theory or hypotheses in the experimentBlind Researcher is unaware of the experiment condition that he/she is administering Researcher is unaware of the experiment condition that he/she is administeringStandardization Experimenter follows a script, and only the script Experimenter follows a script, and only the script “Canned” Experimenter Audio/Video/Print material gives instructions Audio/Video/Print material gives instructions

24 Other Issues with Demand Characteristics Evaluation Apprehension: Subjects know that they are being evaluated and this changes their behavior Subjects know that they are being evaluated and this changes their behaviorSolutions Double-blind experiments Double-blind experiments Experiments in natural setting (i.e., subjects do not know they are in an experiment) Experiments in natural setting (i.e., subjects do not know they are in an experiment) Cover stories Cover stories Hidden measurements Hidden measurements “Faithful Subject” “Faithful Subject”

25 Generalizability (external validity) in Experiments Threats to external validity always involve an interaction of the treatment group with some other factor. Threats usually fall into 3 types: Setting Setting Population Population History History

26 Three threats to generalizability in experiments Setting Physical and social context of the experiment Physical and social context of the experimentPopulation Is there something specific about the sample that interacts with the treatment? Is there something specific about the sample that interacts with the treatment?History Is there something about the time that interacts with the treatment? Is there something about the time that interacts with the treatment?

27 Why Generalizability is not always a problem Experiments often are trying to isolate specific causes and effects in controlled settings. Thus, they may not even be claiming to be generalizable to specific settings. Experimental findings can provide theoretical basis for real-world tests. It is often a balancing act for research: true causation versus large-scale associational and comparative testing.

28 Considerations before using experiments Cost and Effort Is the effort worth it to test the concepts you are interested in? Is the effort worth it to test the concepts you are interested in? Manipulation and Control Will you actually be able to manipulate the key concept(s)? Will you actually be able to manipulate the key concept(s)? Importance of Generalizability Are you testing theory, or trying to establish a real- world test? Are you testing theory, or trying to establish a real- world test?