The Experimental Approach September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Breakdown of an Experiment The Question Previous Work Hypothesis Experimental Design Data Analysis What Can One Conclude? September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

EXAMPLE September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

The Question What issue are we trying to address? How should information be displayed so that it can be best remembered? September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Previous Work 1.Has this topic been examined before? 2.How did they try to answer the questions? –experimental design –analysis methods 3.Were there any problems/limitations? Re-evaluate our original question September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Previous Work Let us say that our literature review on pubmed.gov reveals that the effects of contrast, color scheme, and flashing font on memorization have already been studied. We notice that nobody has studied the effect of font size on memorization yet. We decide to study the effect of font size on memorization. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Hypothesis Intuitively, we think that information may be remembered better when presented in a large font than in a small font. Information shown at a larger visual angle is “seen” by more neurons in the visual system and may thus have a better chance of being remembered. (Notice that this is not a real study or a real argument.)

Experimental Design Experimental Task Dependent Variables Independent Variables Experimental Conditions Trial Sequence Subjects September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

The Experimental Task 0:000:050: Distracting Task _ _ _ _ _ 0:20 TRIAL September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Types of Variables Dependent Variables: variables measured in the experiment: % CORRECT Other Examples: –Resting heart rate –Subjective rating Independent Variables: variables manipulated by the experimenter: FONT SIZE Other Examples: –Hours of running/week –Types of emotion evoking images shown September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Experimental Conditions Each level (value) of the independent variable (or combinations of levels for multiple variables) that we want to test is an experimental condition. In our example, we only have one independent variable: font size. Let us keep things as simple as possible and test only two different levels, i.e., two conditions: small font and large font. We have to include trials for each condition in our experiment. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

0:000:050: Distracting Task _ _ _ _ _ 0:20 Small-font trial: September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach Experimental Trials 0:000:050: Distracting Task _ _ _ _ _ 0:20 Large-font trial:

Trial Sequence All measurements of dependent variables in human subjects are noisy, i.e., their values vary. Therefore, it is important to make a large number of measurements in each experimental condition. Depending on the task, subjects can perform comfortably for 40 to 60 minutes per session. If necessary, multiple sessions per subject can be run. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Trial Sequence In our example, one trial takes 20 seconds. We decide to present 60 trials in each of the two conditions (small font vs. large font). This gives us a total of 120 trials, i.e. 120  20 = 2400 s = 40 minutes. Should we first present the 60 small-font trials, followed by the 60 large-font trials? Or should we use a different sequence? September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Trial Sequence No, first presenting all small-font trials, followed by all large-font trails, is not a good idea. Subjects may improve through practice over the course of the experiment, or they may get tired. If we found better memorization (% correct) for large font than for small font, we cannot be sure that it was actually the font that caused this difference. It is possible that subjects were already better trained once they reached the large-font trials. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Randomization The previous example shows a simple case of a confound – changes in the independent variable being accompanied by changes in other parameters. Confounds prevent us from making sound conclusions about the effect of independent variables on dependent ones and thus must be avoided. In our example, we can avoid the confound by presenting the 120 trials in random order. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Blocked Design In some cases, we do not want to use a completely randomized trial sequence. For example, if subjects have to perform very different tasks across conditions, switching between tasks too often may be disruptive. In that case, we can use blocks of several trials of the same condition, and randomize the order of blocks. If we are using large blocks, we should make sure that we counterbalance the order of blocks across subjects. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Blocked Design In our example (even though it does not require blocking), we could form blocks of 20 trials, resulting in 3 small-font blocks and 3 large-font blocks. We present these blocks in random order and make sure that we distribute conditions in each “time slot” evenly across subjects. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Blocked Design Subject 1 September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach Large-Font Block Small-Font Block Large-Font Block Subject 2 Large-Font Block Small-Font Block Large-Font Block Subject 3 Large-Font Block Small-Font Block Large-Font Block Subject 4 Large-Font Block Small-Font Block Large-Font Block

Subjects In order to draw conclusions about human performance in general, we should test a reasonable number of subjects. We need more subjects if the expected difference in the dependent variable is small or if its expected variance is large. In typical behavioral experiments like our example, about 10 to 20 subjects are appropriate. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Data Analysis For many experimental designs, our hypothesis is confirmed if we find a significant effect, i.e., a difference in the mean value of the dependent variable across conditions in the predicted direction. In our example, let us assume that subjects gave an average of 60% correct responses in the small-font condition and an average of 70% in the large-font condition. Can we confirm our hypothesis, i.e., conclude that larger font leads to better memorization? September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Data Analysis Not yet! The problem is the variance in our measure. Due to this variance, we will get a different result every time we run an experiment, even for the same subjects. Therefore, the difference in mean % correct between the conditions could have occurred coincidentally, without font size having any effect. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Analytical Statistics To ensure that our results are reliable, we have to perform analytical statistics on our data (e.g., t-test, analysis of variance). These tests will tell us the probability p that the difference in a dependent variable across two (or more) conditions is just coincidental. In general, cognitive scientists require that p < 0.05 (probability less than 5%) in order to speak of a significant or reliable effect. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Standard Error (Bars) For visualization of such reliability, researchers typically indicate in charts the standard error of the mean values. The standard error is a measure of how reliable the estimate of the variable’s mean value is. It increases with greater variance in the variable, and it decreases with more measurements being included. Error bars in charts typically indicate standard error. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Standard Error (Bars) Probably no significant effect September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach Probably significant effect

Conclusion If the p-value is below a certain threshold, then we can conclude that our hypothesis was confirmed. We also have to report this p-value to indicate the reliability of our result, and we have to reveal the details of our statistical analysis. If the p-value is above the threshold, then our hypothesis was not confirmed. Unfortunately, most negative results – unless very surprising – are never published. September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach

Types of Experimental Designs Within Subject: Examine Effects within a given subject Between Subjects: Examine effects between groups of subjects Mixed Design: Examine effects within and across subjects September 15, 2009Introduction to Cognitive Science Lecture 3: The Experimental Approach