Biased sample

Editor-In-Chief: C. Michael Gibson, M.S., M.D. [1]

Overview[edit | edit source]

A biased sample is a statistical sample of a population where some members of the population are less likely to be included than others. An extreme form of biased sampling occurs when certain members of the population are totally excluded from the sample (that is, they have zero probability of being selected). For example, a survey of high school students to measure teenage use of illegal drugs will be a biased sample because it does not include home schooled students or dropouts. A sample is also biased if certain members are underrepresented or overrepresented relative to others in the population. For example, a "man on the street" interview which selects people who walk by a certain location is going to have an over-representation of healthy individuals who are more likely to be out of the home than individuals with a chronic illness.

Problems caused by a biased sample[edit | edit source]

A biased sample causes problems because any statistic computed from that sample has the potential to be consistently erroneous. The bias can lead to an over- or under-representation of the corresponding parameter in the population. Almost every sample in practice is biased because it is practically impossible to ensure a perfectly random sample. If the degree of underrepresentation is small, the sample can be treated as a reasonable approximation to a random sample. Also, if the group that is underrepresented does not differ markedly from the other groups in the quantity being measured, then a random sample can still be a reasonable approximation.

The word bias in common usage has a strong negative connotation, and implies a deliberate intent to mislead. In statistical usage, bias represents a mathematical property. While some individuals might deliberately use a biased sample to produce misleading results, more often, a biased sample is just a reflection of the difficulty in obtaining a truly representative sample.

Some samples use a design that is deliberately biased. The U.S. National Center for Health Statistics will deliberately oversample from minority populations in many of its nationwide surveys in order to gain sufficient precision for estimates within these groups^{(NCHS 2007)}. These surveys require the use of sample weights (see below) to produce proper estimates across all racial and ethnic groups.

Examples of biased samples[edit | edit source]

Online and phone-in polls are biased samples because the respondents are self-selected. Those individuals who are highly motivated to respond, typically individuals who have strong opinions, are overrepresented, and individuals that are indifferent or apathetic are less likely to respond. This often leads to a polarization of responses with extreme perspectives being given a disproportionate weight in the summary. As a result, these types of polls are regarded as unscientific.

A classic example of a biased sample and the misleading results it produced occurred in 1936. In the early days of opinion polling, the American Literary Digest magazine collected over two million postal surveys and predicted that the Republican candidate in the U.S. presidential election, Alf Landon, would beat the incumbent president, Franklin Roosevelt by a large margin. The result was the exact opposite. The Literary Digest survey represented a sample collected from readers of the magazine, supplemented by records of registered automobile owners and telephone users. This sample included an over-representation of individuals who were rich, who, as a group, were more likely to vote for the Republican candidate. In contrast, a poll of only 50 thousand citizens selected by George Gallup's organization successfully predicted the result, leading to the popularity of the Gallup poll.

Another classic example occurred in the 1948 Presidential Election. On Election night, the Chicago Tribune printed the headline DEWEY DEFEATS TRUMAN, which turned out to be mistaken. In the morning the grinning President-Elect, Harry S. Truman, was photographed holding a newspaper bearing this headline. The reason the Tribune was mistaken is that their editor trusted the results of a phone survey. Survey research was then in its infancy, and few academics realized that a sample of telephone users was not representative of the general population. Telephones were not yet widespread, and those who had them tended to be prosperous and have stable addresses. (In many cities, the Bell System telephone directory contained the same names as the Social Register.)In addition, the Gallup poll that the Tribune based its headline on was over two weeks old at the time of the printing.^[1]

Statistical corrections for a biased sample[edit | edit source]

If entire segments of the population are excluded from a sample, then there are no adjustments that can produce estimates that are representative of the entire population. But if some groups are underrepresented and you can quantify the degree of underrepresentation, then sample weights can correct the bias.

For example, a hypothetical population might include 10 million men and 10 million women. Suppose that a biased sample of 100 patients included 20 men and 80 women. A researcher could correct for this imbalance by attaching a weight of 2.5 for each male and 0.625 for each female. This would adjust any estimates to achieve the same expected value as a sample that included exactly 50 men and 50 women.

Spotlight fallacy[edit | edit source]

The Spotlight fallacy is committed when a person uncritically assumes that all members or cases of a certain class or type are like those that receive the most attention or coverage in the media. This line of “reasoning” has the following form:

1. Xs with quality Q receive a great deal of attention or coverage in the media. 2. Therefore all Xs have quality Q.

This line of reasoning is fallacious since the mere fact that someone or something attracts the most attention or coverage in the media does not mean that it automatically represents the whole population. For example, suppose a mass murderer from Old Town, Maine, received a great deal of attention in the media. It would hardly follow that everyone from the town is a mass murderer.

The Spotlight fallacy derives its name from the fact that receiving a great deal of attention or coverage is often referred to as being in the spotlight. It is similar to Hasty Generalization, Biased Sample and Misleading Vividness because the error being made involves generalizing about a population based on an inadequate or flawed sample. The Spotlight Fallacy is a very common fallacy. This fallacy most often occurs when people assume that those who receive the most media attention actually represent the groups they belong to. For example, some people began to believe that all those who oppose abortion are willing to gun down doctors in cold blood simply because those incidents received a great deal of media attention. Since the news media typically cover people or events that are unusual or exceptional, it is somewhat odd for people to believe that such people or events are representative.

Examples[edit | edit source]

1. I wouldn't like to go to America because of all the gun crime; we see it on the news all the time.
2. Muslims are always in the news blowing other people up; so all Muslims are terrorists.
3. Doctor: Why don't patients make some effort to look after themselves? My surgery is full of people who eat, drink, smoke and don't get any exercise. Of course he may have many more patients who do look after themselves and don't often turn up in his surgery.
4. Why do young people all take drugs and go around mugging old ladies? You read about it in the paper all the time!
5. Child: When I grow up I want to be a singer. Have you seen how much money those pop-stars make?!

See also[edit | edit source]

• File drawer problem

References[edit | edit source]

National Center for Health Statistics (2007). Minority Health. http://www.cdc.gov/nchs/about/otheract/minority/minority.htm

Template:Jb1

Template:WikiDoc Sources