1. Christen Madsen II, Director of Quantitative Research Consulting Center at the Graduate Center in the inaugrual year, a Provost’s Office INitiative
Aysenur Benevento, Doctoral Candidate in Developmental Psychology
Teresa Ober We

2. Operationalizing Violence on TV
(Courtesy of Madelynn Shell via STP Facebook Group 1/22/14)
In today’s world we are very concerned about violence in children’s lives.
How much violence are children exposed to via cartoons?
Rabbit Season (
Count the number of violent acts
What number did you come up with?

3. What is missing?

4. What is missing? Students come up with different numbers.
Need an operational definition for violent acts.
Categories: physical, verbal, weapons
Create a score sheet.
Track behavior one subject at at time.

5. What is missing? Key Term: Operationalizing Constructs
Students come up with different numbers.
Need an operational definition for violent acts.
Categories: physical, verbal, weapons
Create a score sheet.
Track behavior one subject at at time.
Key Term: Operationalizing Constructs
Defining a fuzzy concept to make the concept clearly measurable.

6. Mnemonic for Scientific Method
Hypothesis
Operationalizing
Measure
Evaluation
Report
Lakin, J. L., Giesler, R. B., Morris, K. A., & Vosmik, J. R. (2007). HOMER as an acronym for the scientific method. Teaching of Psychology, 34(2),

7. Thinking about Probabilities / Computing Odds Ratios

8. Availability Heuristic
People estimate frequencies or probabilities in terms of how easy it is to think of relevant examples.
Availability Heuristic
This is based on an activity from a introductory cognitive psychology course that I taught in the past. It is used to teach concepts related to a common heuristics we use, namely the availability heuristic. We use the availability heuristic when we estimate frequency or probability in terms of how easily we can think of examples of something. This heuristic is generally accurate in our daily lives, and people can estimate relative frequency with impressive accuracy.
This demonstration also provides a teachable moment for students by showing them that we should be especially conscious of the availability heuristic when making judgments based on things we hear, whether it is from someone we know, the news, etc. That’s why I included this schematic, not based on any actual data of course, to show how little of what is happening around us is actually reported in the news.
So while there is a convenience of choosing types of news and news sources that you have readily presented to you at your leisure, one could argue that it allows us to construct a narrow version of current events.
So let’s see how it actually works...

9. Which is More Common? There are more words that begin with R
There are more words that have R as their third letter
Both “a” and “b” are about the same (within 5% of each other).
To demonstrate the availability heuristic, here is a quick question for you all.
Which is more common:
Words that begin with the letter “R” such as red, rose, and so forth
Words that have “R” as the third letter such as car, far, etc.
Or both “A” and “B” about the same or within about 5% of each other
Adapted from Tversky and Kahneman (1973)

10. R as 1st letter (2386) vs 3rd letter (4247)
Which is More Common?
There are more words that begin with R
There are more words that have R as their third letter
Both “a” and “b” are about the same (within 5% of each other).
Frequency data:
R as 1st letter (2386) vs 3rd letter (4247)
The answer as some of you may have guessed was that there are more words that begin with the letter R, which is actually incorrect. In reality, there are nearly twice as many words that have the letter “R” in the third position as opposed to the first position.
But why does it seem that this is incorrect? Probably because you are more familiar with words that begin with the letter R. It comes to your mind more easily than thinking of words that have R in as the third letter of the word.
Adapted from Tversky and Kahneman (1973)

11. Which is More Fatal? Stroke or All accidents Tornado or Asthma
Here is another example.
For this second demonstration, try to guess which one is more likely to lead to a fatality. Which of each pair is more likely cause a fatality? If this were done in class, students would normally have a chance to discuss what they believed to be the correct answer, and were encourage to explain their reasoning.
Adapted from Slovic, Fischhoff, and Lichtenstein (1976)

12. Which is More Fatal? Stroke (1.85 to 1) or All accidents
Tornado or Asthma (20.90 to 1)
This one involves judging the frequency of deaths due to certain causes. People tend to overestimate the number of deaths from, a tornado, but underestimate the number of deaths from something like asthma. This is because we are probably more likely to hear about deaths from tornadoes in the news, so such instances bring to mind a greater number of examples. Meanwhile, in the absence of examples of fatalities due to asthma, we might be inclined to underestimate how fatal it can actually be.
It is likely that several students will use the availability heuristic (e.g., the frequency of media reports) to judge the frequency of deaths. In the 1970’s, Slovic, Fischhoff, & Lichtenstein (1976) discovered that only a small percentage of participants gave the correct answers:
20% answered that strokes are more common than all accidents
42% answered that asthma is more common than tornado
Other examples include
Diabetes v. breast cancer (23% answered incorrectly that diabetes is more common than breast cancer)
Stomach cancer v. lung cancer (25% answered incorrectly that stomach cancer is more common than lung cancer)
Appendicitis v. pregnancy (17% answered incorrectly that appendicitis is more common than pregnancy)
Has increased media reporting on some of these causes of deaths (e.g., diabetes) changed student estimates since this study was conducted first in 1976? Perhaps. Nevertheless, the effect can still be demonstrated with more updated examples.
Adapted from Slovic, Fischhoff, and Lichtenstein (1976)

13. Using Intuition to Estimate Frequencies
“Contaminants”
Recency
Familiarity
Availability
Estimated Frequency
How can we explain this phenomenon to our students? Here is one model of how the intuitive system works. You have certain things that are just available to you and you interpret them to infer an estimated frequency.
This is very similar to system 1 according to the theories of Kahneman and Tversky. Now, we are going to learn about system 2, which is more methodical.
Since it is not possible to know everything that goes on around us all time, nor would such information necessarily be relevant to our lives, the availability heuristic can actually be quite useful sometimes in gauging the likelihood that something is to occur. However, availability may be contaminated by two factors that are not related to objective frequency: recency and familiarity. Therefore, when you make frequency judgments, ask yourself whether you are giving a special advantage to items that occurred more recently or that are somehow more familiar.
Recency: Memory is better for more recent items. Recent items are more available. People judge recent items to be more likely than they really are.
Familiarity: Events with which we are highly familiar lead us to overrate their likelihood relative to events with which we are less familiar.
But how to get students to be more conscious of the incorrect conclusions we can come to as a result of contaminants affecting our use of the availability heuristic? One way is to get them to consciously strive to observe true frequencies and question their beliefs and knowledge based on available information.
Adapted from Matlin and Farmer (2015)

14. Using Odds Ratios to Estimate Frequencies
Is the train more often delayed when you are running late?
Week-long “diary” study
Each time the student rides the train, they record whether they were running late and whether the train was delayed
Create a worksheet (to be collected as homework)
The availability heuristic can also create illusory correlations, when two variables appear to be correlated, although there is no statistical relationship.
For example:
The weather is always bad on the weekend.
The train is always late when you are running behind.
The phone always rings when you are busy.
Preventing this phenomenon can be done simply by calculating an odds ratio, but most people don’t bother to do this. If students did, they might be more aware of the types of illusory correlations they actually believe in as a result of an overactive availability heuristic. Here is a simple activity to get students thinking about this.
In groups, students choose a perceived correlation and set out to record it using a contingency table (such as the one below)
During the next class, or up to a week later, students briefly report their findings.
In so doing, they discuss whether the perceived relationship be due to an actual correlation? If so, what might be the relationship between variables?
Or whether the perceived relationship may be due to contaminants in the availability heuristic? If so, was it due to either recency or familiarity and what do they take as evidence of this?
In this hypothetical example, if students decided to keep track of the number of times the train arrived late when they were running behind, they would find that there is no relationship. In fact, the odds of a train running late when they are or are not running behind is exactly the same. The odds is 3:2.

15. Step 1: Tabulating the Data
Train Delayed
Train Not Delayed
Odds Ratio
Running Late
120 74
Not Running Late 33 20
What can we conclude from looking at this raw data?
Here is some simulated data for this example. Let’s say that across the entire class, students reported that the train was delayed when they were running late about 120 times over the course of the week, it was not delayed when they were late about 74 times, it was delayed when they were running late about 33 times, and it was neither delayed nor were they running late during 20 observed instances.
Looking at this data, what might we conclude?
Do we really know anything about relative frequencies yet? No, not really. So we cannot really conclude much from the raw counts.

16. Step 2: Calculating Odds Ratios
Train Delayed
Train Not Delayed
Odds Ratio
Running Late
120 74
1.62 to 1
Not Running Late 33 20
1.65 to 1
Calculate proportions: 120/74 = 1.62
33/20 = 1.65
However, we can use this data to create odds ratios, or the relative frequency of two possible outcomes for each group: either running late or not running late.
In determining the odds ratios, we first have to calculate the proportions for delayed over not delayed for each group (either running late or not running late).
When we do that we find that the likelihood of a train being delayed to not delayed when you are running late is 1.62 to 1. This means that for each one instance that the train is not delayed, it is likely to be delayed 1.62 times. In other words, it is more likely to be delayed than not when you are running late.
We do the same for the times for the not-running-late group and find an odds ratio of 1.65.

17. Step 2: Calculating Odds Ratios
Train Delayed
Train Not Delayed
Odds Ratio
Running Late
120 74
1.62 to 1
Not Running Late 33 20
1.65 to 1
Is the train more often delayed when you are running late?
So now how do judge the relative likelihood now? We see that it is only slightly more likely to occur when students are not running late.
However, we still know nothing about whether these are statistically significant. Fortunately, there are statistical tests for this.

18. Different Modes for Frequency Estimation
“Thinking Fast and Slow”
(Kahneman, 2011)
Here is another final take-away for students, which shows a comparison of the System 1 and System 2 of cognition proposed by Kahneman and Tversky’s work. Here you see that System 1 consists of a fast, unconscious, automatic and generally intuitive system, while System 2 consists of a slower, more conscious and controlled system.
While the availability heuristic is greater for making snap-second decisions about things, it can often lead us astray. We we train students and ourselves the tools to utilize system 2, they can in turn make better decisions not just about cognitive phenomenon, but also about everyday events.

19. Graphing the Serial Position Effect

20. Free Recall Task You hear a list of 20 words to remember.
(one word at a time)
Try to remember them to the best of your ability.
Do not write them down.
(See excel sheet for graphing data.)
House
Bicycle
Jacket
Banana
Couch
Tree
Carrot
Spoon
School
Radio
Guitar
Office
Blanket
Kite
Sneakers
Letter
Candle
Ice Cream
Truck
Window

21. Collecting and Using Student-Generated Datasets

22. Getting data from students
Incorporate data collection as homework
Model data collection in class
Ratings
Observation studies
Freelists

23. Data management
Cleaning
Formatting
Re-structuring

24. Data visualizations in Excel
Pivot tables

25. Data visualizations that work
What makes a good chart?
Should be able to understand the point without context

26. Data visualizations that work

27. Data visualizations that don’t work

28. Simple inferential statistics
Use functions in Excel. Returns p-values
T-tests
Correlations

29. Conclusions

30. Statistical Literacy: Defined
“‘Statistical Literacy’ is the ability to understand and critically evaluate statistical results that permeate our daily lives-coupled with the ability to appreciate the contributions that statistical thinking can make in public and private, professional and personal decisions.”
Wallman (1993)

31. Developing Habits of Quantitative Reasoning
Introduce quantitative reasoning from the beginning of instruction
Provide contexts for use of statistics, rather than focusing on abstract concepts
Encourage students to be skeptical
Consider a range of variables in relation to the complexity of psychological phenomena
Relate data to specific research questions
Go beyond the textbook to facilitate meaningful and varied uses of data collection and analysis
Chance (2002)

32. Additional Resources Some resources with open-source data:
Some published classroom activities to promote quantitative reasoning:
PsychBusters Activity (Blessing & Blessing, 2010)
Detective Work on Statistics Street (Zeedyk, 2006)