Research Methods Review Advanced Cognitive Psychology PSY Fall, 2004

Overview The Basics The Basics Experimental Methods Experimental Methods Designs Designs Within-subjects Within-subjects Between-subjects Between-subjects Factorial Factorial Statistical Review Statistical Review Example of Experiment Example of Experiment Don’t let me forget a simple assignment to give you before you leave Don’t let me forget a simple assignment to give you before you leave

What are Research Methods? Application of the scientific method to studying behavior Application of the scientific method to studying behavior Scientific Method = observe, hypothesize/predict, test, conclude Scientific Method = observe, hypothesize/predict, test, conclude Work with structured observations Work with structured observations Deal with solvable problems Deal with solvable problems Produce publicly verifiable information Produce publicly verifiable information Hypotheses, theories, and explanations MUST BE FALSIFIABLE Hypotheses, theories, and explanations MUST BE FALSIFIABLE

General Methods – The Basics Watch people in public settings and drawing conclusions = Observational/Descriptive Watch people in public settings and drawing conclusions = Observational/Descriptive Conducting a survey, questionnaire, interview, poll, etc. and comparing responses on one question to other questions = Relational/Correlational Conducting a survey, questionnaire, interview, poll, etc. and comparing responses on one question to other questions = Relational/Correlational Randomly forming 2 or more groups, treat them all differently, see how the outcomes differ = Experimental Randomly forming 2 or more groups, treat them all differently, see how the outcomes differ = Experimental Compare two or more groups (that are not randomly formed and are different to begin with) in a variety of ways = Quasi-Experimental Compare two or more groups (that are not randomly formed and are different to begin with) in a variety of ways = Quasi-Experimental

Experimental Method This type of method attempts to answer questions about CAUSE This type of method attempts to answer questions about CAUSE Variables Variables Independent = manipulated or changed for various subjects; IV Independent = manipulated or changed for various subjects; IV Dependent = the way to measure a change in behavior (or not change); DV Dependent = the way to measure a change in behavior (or not change); DV Control = a potential IV that is held constant at one level – all subjects are exposed to that one level Control = a potential IV that is held constant at one level – all subjects are exposed to that one level Manipulation = change Manipulation = change

Manipulating Variables - Designs One variable – multiple levels One variable – multiple levels Univariate or one-way design Univariate or one-way design Level = one aspect of the variable; condition Level = one aspect of the variable; condition Example Example Manipulation Types Manipulation Types Between Subjects = ONE Between Subjects = ONE To expose some subjects to one level of the IV and other subjects to another level of the IV – subjects are not exposed to all aspects of the variable To expose some subjects to one level of the IV and other subjects to another level of the IV – subjects are not exposed to all aspects of the variable Within Subjects = ALL Within Subjects = ALL To expose all subjects to all the levels of the IV To expose all subjects to all the levels of the IV

The BOX Boxes with rows and columns – essential for understanding experiment design and statistics Boxes with rows and columns – essential for understanding experiment design and statistics Level 1 Level 2Level 1 Level 2 Variable A Variable B Row 1 Row 2 Column 2Column 1

Factorial Designs When more than one variable is manipulated in an experiment When more than one variable is manipulated in an experiment 2 Variables = Two way 2 Variables = Two way 3 or more variables = multivariate 3 or more variables = multivariate Between subjects design = all variables are manipulated between/across the levels Between subjects design = all variables are manipulated between/across the levels Within subjects design = all subjects receive all the levels of all the variables Within subjects design = all subjects receive all the levels of all the variables Mixed design = at least one variable is manipulated between subjects and at least one variable is manipulated within subjects Mixed design = at least one variable is manipulated between subjects and at least one variable is manipulated within subjects

Statistical Review Populations and Samples Populations and Samples Hypothesis Testing Hypothesis Testing Using methods and statistical tests together Using methods and statistical tests together

Population vs. Sample Population = everyone that you are interested in studying or at the very least, generalizing your results to Population = everyone that you are interested in studying or at the very least, generalizing your results to Ex: Women; children; Men over 40 Ex: Women; children; Men over 40 You can’t possibly hope to study the entire population You can’t possibly hope to study the entire population Sample = a subset of the population that contains the important characteristics of the population; a sample is representative of its population Sample = a subset of the population that contains the important characteristics of the population; a sample is representative of its population Various techniques for sampling from a population Various techniques for sampling from a population Ex: PSU Women; children at BFC; Male faculty in psychology over 40 Ex: PSU Women; children at BFC; Male faculty in psychology over 40 Why does this matter? Why does this matter? Allows research to occur without the impossible task of studying everyone Allows research to occur without the impossible task of studying everyone Important assumptions for statistics Important assumptions for statistics

Hypothesis Testing Comparing the Null and Experimental hypotheses to predict the likelihood of one being show to be true Comparing the Null and Experimental hypotheses to predict the likelihood of one being show to be true Null Hypothesis = There is no difference; nothing will change; zero Null Hypothesis = There is no difference; nothing will change; zero Experimental Hypothesis = There will be a difference; something will change; non- zero Experimental Hypothesis = There will be a difference; something will change; non- zero

Hypothesis Testing Null Hypothesis Null Hypothesis There is no difference in GRE scores between males and females Experimental Hypothesis Experimental Hypothesis There is a difference in GRE scores between males and females In the real world, the Null is Your decision TrueFalse Reject H 0 Type I error Correct Decision Retain H 0 Correct Decision Type II error

Hypothesis Testing Example Barbie and Kendall – Chocolate Eaters (to be read in class) From this, what could we conclude about this contradiction? According to our assumption, in the real world, H0 is true. Therefore, if Barbie rejected H 0 because she thought it was wrong (based on her study's results), what has happened? Did she commit an error or make a correct decision? If Barbie would have retained H 0 (and to do that, her study would have resulted in no differences between the mean exam scores from the two groups), and we assume that in the real world, H 0 is true, did she commit an error or make a correct decision? If we now assume that Kendall is wrong and in the real world, H 0 is false, and Barbie rejected H 0 because she thought it was wrong (based on her study's results), what has happened? Did she commit an error or make a correct decision? Again, assume that Kendall is wrong, and H 0 is false in the real world. If Barbie would have retained H 0 (and to do that, her study would have resulted in no differences between the mean exam scores from the two groups), and we assume that in the real world, H 0 is true, did she commit an error or make a correct decision?

Hypothesis Testing AlphaBetaEffect SizeStatistical Power Definition The probability of committing a Type I error The probability of committing a Type II error The size of the effect (difference/ relationship) The probability of rejecting a false null hypothesis (or the probability of finding an effect if one exists) When to use Set prior to collecting data Based on how much power you want (determined before collecting data) or how much power you actually have (determined after collecting data) To be determined every time you run an inferential statistic (correlation, t-test, ANOVA, chi- square) Prior to collecting data: estimate power and effect size to determine how many subjects you need to achieve certain level of power. After analyzing data: determine effect size and combine that with sample size to determine power Interpre- tation Percent chance of actually committing a Type I error (if the null is true) Percent chance of actually committing a Type II error (if the null is false) Small, medium, or large (quantitative values depend on type of effect size test you used) Percent chance that you will find a statistically significant result given your sample size (N) and effect size (and assuming the null is false) Example α =.05 means a 5% chance of committing a Type I error β =.20 means a 20% chance of committing a Type II error d =.50 means a medium effect size for the effect size corresponding to a t-test Power =.80 means an 80% chance of finding a statistically significant result given your N, ES, and assumption that the null is false

Scales of Measurement Nominal = used to identify a particular characteristics of the scale; also called categorical (categories are mutually exclusive) Nominal = used to identify a particular characteristics of the scale; also called categorical (categories are mutually exclusive) EX: Sex (M/F); ZIP Codes EX: Sex (M/F); ZIP Codes Ordinal = numbers indicate whether there is more or less of the measured variable; order is important Ordinal = numbers indicate whether there is more or less of the measured variable; order is important EX: Levels of education (Freshman, Sophomore, Junior, Senior); Olympic medals EX: Levels of education (Freshman, Sophomore, Junior, Senior); Olympic medals Interval = numbers correspond exactly to changes in the measured variable and there are equal distances between numbers that correspond to equal changes in the measured variable Interval = numbers correspond exactly to changes in the measured variable and there are equal distances between numbers that correspond to equal changes in the measured variable EX: IQ; Temperature (Fahrenheit, Celcius) EX: IQ; Temperature (Fahrenheit, Celcius) Ratio = like an interval scale (equal intervals) but also includes a true zero point (the absence of the measured variable). This allows for multiplication and division of scale values. Ratio = like an interval scale (equal intervals) but also includes a true zero point (the absence of the measured variable). This allows for multiplication and division of scale values. EX: Weight; Height; Temperature (degrees Kelvin) EX: Weight; Height; Temperature (degrees Kelvin)

Descriptive Statistics Decide how to summarize and represent data based on the TYPE of data that you have (its scale of measurement) Measures of Central Tendency Measures of Central Tendency 1. Nominal Scale – Mode 2. Ordinal Scale – Median 3. Interval/Ratio Scales - Mean Measures of Dispersion Measures of Dispersion 1. Nominal Scale – Range 2. Ordinal Scale – Absolute Deviation from the Median 3. Interval/Ratio Scales – Variance, Standard Error, Standard Deviation Graphical Representations of Data Graphical Representations of Data 1. Nominal/Ordinal Scales (Qualitative Data) – Bar Graph, Pie Chart 2. Interval/Ratio Scales (Quantitative Data) – Line Graph, Frequency Polygon, Histogram

Inferential Statistics Decide which test to use based on the TYPE of data you have and the KIND of outcome you are looking for Relationships (Correlations) Relationships (Correlations) 1. Nominal Scale – Chi-Square Test of Independence, or Phi 2. Ordinal Scale – Kendall’s Tau or Spearman’s r 3. Interval/Ratio Scales – Pearson’s r

Differences Differences 1. One Independent Variable Between-Subjects manipulation Between-Subjects manipulation 2 levels 2 levels Nonparametric – Chi-Square Goodness of Fit (nominal) and Mann Whitney U (ordinal) Nonparametric – Chi-Square Goodness of Fit (nominal) and Mann Whitney U (ordinal) Parametric – Independent means t-test or one-way ANOVA Parametric – Independent means t-test or one-way ANOVA 3+ levels 3+ levels Nonparametric – Chi-Square Goodness of Fit (nominal) and Kruskal Wallace (ordinal) Nonparametric – Chi-Square Goodness of Fit (nominal) and Kruskal Wallace (ordinal) Parametric – One-way ANOVA Parametric – One-way ANOVA Within-Subjects manipulation Within-Subjects manipulation 2 levels 2 levels Nonparametric – Chi-Square Goodness of Fit (nominal) and Mann Whitney U (ordinal) Nonparametric – Chi-Square Goodness of Fit (nominal) and Mann Whitney U (ordinal) Parametric – Independent means t-test or one-way ANOVA Parametric – Independent means t-test or one-way ANOVA 3+ levels 3+ levels Nonparametric – no good tests Nonparametric – no good tests Parametric – One way Repeated-Measures ANOVA Parametric – One way Repeated-Measures ANOVA

Differences, continued Differences, continued 2. Two Independent Variables Between-Subjects manipulation Between-Subjects manipulation Nonparametric – Wilcoxon-Wilcox and Friedman tests Nonparametric – Wilcoxon-Wilcox and Friedman tests Parametric – Factorial ANOVA Parametric – Factorial ANOVA Within-Subjects manipulation Within-Subjects manipulation Nonparametric – Friedman test Nonparametric – Friedman test Parametric – Factorial Repeated- Measures ANOVA Parametric – Factorial Repeated- Measures ANOVA Mixed manipulation – Mixed Factorial ANOVA Mixed manipulation – Mixed Factorial ANOVA

Putting this all together… To study behavior, we have to create conditions that are controlled enough to be able to predict an outcome in the controlled conditions To study behavior, we have to create conditions that are controlled enough to be able to predict an outcome in the controlled conditions We have to think about how to study the behavior of interest and how to make it change in a predictable fashion We have to think about how to study the behavior of interest and how to make it change in a predictable fashion Experimental methodology allows researchers to control the sample and expose the participants to changes that are predicted to influence the outcome Experimental methodology allows researchers to control the sample and expose the participants to changes that are predicted to influence the outcome When you construct a particular experiment or use a particular research method, a certain logic applies when choosing the “proper” statistical method to analyze the results/outcome When you construct a particular experiment or use a particular research method, a certain logic applies when choosing the “proper” statistical method to analyze the results/outcome

Experiment Example This will be shown in class This will be shown in class Signal Detection Experiment Signal Detection Experiment IV – Presence of Target IV – Presence of Target Levels: Present or Absent Levels: Present or Absent Manipulation: Within-Subjects Manipulation: Within-Subjects Measure/DV: Hits, False Alarms, Correct Rejections, Misses Measure/DV: Hits, False Alarms, Correct Rejections, Misses