Unfair Sampling Let's look at some realistic cases where unfair sampling might lead to false conclusions.

The danger of air travel: Many people don't like to fly because they feel like it is dangerous. Part of the reason is probably because they see newspaper articles about plane crashes and feel that this is a relatively common event. What seems to happen in our minds is that we compare the number of crashes we hear about with an intuitive estimate of how many don't crash. The trouble is, we don't hear much about the planes that don't crash, while we hear, often in great detail, about the ones that do, even when the crashes occur on the other side of the world. We may mainly know about safe landings when we or our friends travel, and occasionally when a celebrity is shown arriving at an airport in the news. So the sampling we are aware of is extremely biased. We are aware of virtually every disaster but only an extremely tiny percentage of the non-disasters. Our natural mental estimation is that flying is far more dangerous than it actually is.

Crimes committed by released prisoners: Occasionally we hear a news story that some person recently paroled from prison who then commits a murder or some other heinous crime. As a result, many people clamor for changes in prison policy to prevent people from being released on parole. The problem is that people are likely to greatly overestimate the percentage of parolees who commit serious crimes because the news media rarely report the cases of people who are paroled and don't commit crimes. Obviously it is undesirable to keep people in prison, at public expense, when they would not cause further problems for society. To make intelligent decisions about public policy we need to know the actual percentage of people released who cause problems; we cannot trust our intuitive impression we get from news stories that don't reflect a fair sample of people who are released from prison.

The danger of terrorist attacks: In some years there have been terrorist attacks on tourists in Europe. If we read that, say, eight tourists have been killed in three separate incidents during the past few months, we might feel that it would not be a good idea to travel to Europe until things have settled down. We might also see comments in the media that this is not a safe time for European travel. However we hear about those tourists that were killed, but most of us know nothing about the millions of tourists who were not killed. As a result out mental perception is that a significant percentage of tourists are being killed, while in fact the percentage is very tiny, probably far less than the number that die of heart attacks or other natural causes while on vacation.

Superstitions based on selective memory: There are many superstitions about events that bring good or bad luck, such as the idea that finding a four-leaf clover is good luck. Part of the reason these superstitions sometimes seem to work is the result of the vagueness of the prediction combined with biased sampling. After finding a four-leaf clover, a variety of things happen to us. If we are eager to believe in the luck of the clover, we can ignore all the things that happen that aren't good and select something that happens that is better than expected. Even if nothing particularly good happens, we may forget that the clover failed but remember other cases where finding a four-leaf clover was indeed followed by some unusual good fortune. If we selectively remember the cases that work and ignore the cases that don't, we can get the impression that the superstition is confirmed by experience.

Predictions with no time limit: Psychics and fortune tellers frequently make predictions that have no time limit associated with them, for example that you will have financial success or start a new romantic relationship. If we keep track of successes and failure for this type of prediction, we will find occasional successes and no failures. The reason there are no failures is that there is always the possibility that the prediction will come true at a later time. As a result, our sampling of the cases by which we judge the psychic always includes the successes and never includes the failures! Obviously this is not a fair sampling. Even when we might reasonably assume a time limit, it is easy to forget predictions that fail while remembering those that succeed.

Movie endorsements: I've seen commercials for newly released movies that show people coming out of the theatre talking about how great the movie was. While it is possible these are actors, it is likely that they are unpaid patrons giving their honest opinion. That does not mean that the average person likes the movie, though. The crew filming the commercial would certainly interview a large number of people and then only include the reactions that are most favorable. Since the selection of cases to be presented is made by the advertising agency, it is clear that the cases shown will not be an unbiased cross-section.

Successful people: I've seen celebrities and other prominent people interviewed on television and they'll often say something like "You've got to take risks" or "You have to pursue your dream". It may be that most people who are huge successes take this attitude. On the other hand, many people who take this attitude may wind up being failures. The failures are never interviewed. The sample contains only the successes. A closely related problem is all the attention given to famous entertainers and athletes. Only a tiny percentage of people who choose a career in sports or entertainment actually become famous, but in the sample we see, all are famous, so we are likely to greatly overestimate the extent to which these are wise career choices.

Whenever a claim of some general principle is important enough to merit examination, we should immediately ask whether the sample of data that led to that conclusion was a fair sample. We should be equally tough on ourselves if we think we have noticed some generality that would be of interest to others. If we care about the truth we have to be on guard against the problems of unfair sampling.