Survey design Stat 472

Questions as Measures What is a Good Question?

In the social sciences, important measurements are based on a question-and-answer process e.g., Children answer questions in standardized tests, and the results are used as measures of their intelligence. The unemployment rate is a calculation of answers about employment status and desire to be in the workforce. Studies of medical outcomes rely on patients’ answers to questions about their health status and quality of life.

There is an almost limitless body of desirable and useful information that can be gathered only by asking people questions. In some cases, that is because we want to know facts that are difficult to observe systematically. e.g., Some crimes are reported to police, many are not. The best way to estimate the rate at which people are victims of crime is to ask a sample of people about their victimization experiences.

We also are often interested in measuring phenomena that only individuals themselves can realized: what people think or know, or what they feel. Our notion is: How to turn the questions and answers into accurate measurement. There are several implications to do that

1. We are not interested in the answers for their own sake 1. We are not interested in the answers for their own sake. We are interested in what the answers tell us about something else. As a result, one critical standard for a good question-and-answer process is that it produced answers that provide meaningful information about what we are trying to describe. 2. The purpose of measurements usually is to produce comparable information about many people or events. Hence, it is important that the measurement process, when applied repeatedly, produces consistent results.

What is a good question? A good question is one that produces answers that are reliable and valid measures of something we want to describe. Reliability is used in the classic Psychometric sense of the extent to which answers are consistent; when the state of what being described is consistent, the answers are consistent Validity, in turn, is the extent to which answers correspond to some hypothetical “true value” of what we are trying to describe or measure.

Regardless of the mode of data collection or the subject matter, there are some standards for questions and some common principles about how to meet those standards.

1. Questions need to be consistently understood