MAT 1000 Mathematics in Today's World

Last Time 1.Collecting data with experiments 2.Practical problems with experiments

Today Measurement Validity Error Bias Variability

Measurement Recall structure of data: individuals and variables Measurement is the assignment of a value to a variable.

Measurement Example Unemployment rate Individuals: working age American adults Variable: employment status Values: employed or unemployed. To measure, ask an individual: “are you employed?”

Measurement Example Heights of MAT 1000 students Individuals: MAT 1000 students Variable: height Values: numeric To measure, use a measuring tape, yard stick, etc.

Measurement An instrument is a device or method for taking a measurement Measuring height: different possible instruments, like measuring tape or yard sticks Unemployment: the instrument is the question asked

Measurement Some instruments give measurements in “units.” Tape measure could give height measured in inches, or in centimeters.

Validity Are we really measuring the attribute we are interested in? Example University admission boards want to measure how prepared applicants are for college. One way this can be measured is with standardized test scores (SAT or ACT score)

Validity A valid measurement is an appropriate representation of the property being measured. An extreme example: measure college readiness by height. Clearly not valid.

Validity But are ACT test scores even a valid measurement of college readiness? Some people argue that they are not—that ACT scores don’t tell us if someone is ready for college.

Validity This is a common scenario. The question we are interested in is somewhat vague, and we have to try and find an appropriate (valid) way to measure. How else could we measure “college readiness”?

Predictive Validity Can we get evidence that a particular measurement like ACT scores is valid? Yes. We look if ACT scores can predict how well an applicant will do in college. If a measurement can predict success on a related task, it has predicitive validity.

Predictive Validity If applicants with ACT scores tend to do well in college (i.e. high GPA), this gives evidence that the ACT is a valid measurement of college readiness. Conclusion: one way to decide if a measurement is valid is to ask if it has predictive validity. How else can we decide?

Rates Measuring highway safety. One method is to count deaths from car accidents. In 1994 there were 40,676 traffic fatalities in the U.S. In 2003 there were 42,643. Was driving becoming less safe?

Rates What else changed from 1994 to 2004? Higher population: 260 million to 290 million More drivers: 175 million to 196 million More miles driven: 2,359 billion to 2,890 billion So is it valid to compare highway safety from 1993 to 2003 using the number of traffic fatalities? No.

Rates A more valid measurement is to use a rate, like traffic fatalities per number of drivers, or traffic fatalities per mile drive. 1993: fatalities per 100,000 drivers 2003: fatalities per 100,000 drivers 1993: 1.7 fatalities per 100,000,000 miles 2003: 1.48 fatalities per 100,000,000 miles

Rates 2012 (most recent year available): 33,561 fatalities, compared to 42,643 in Are we safer on the road? Yes, but we have to use rates to see this Even though the count is lower in 2012, that doesn’t make it a valid measurement.

Rates Rates from 2003 to 2012: 2003: fatalities per 100,000 drivers 2012: fatalities per 100,000 drivers 2003: 1.48 fatalities per 100,000,000 miles 2012: 1.13 fatalities per 100,000,000 miles

Validity Is a measurement valid? Does it have predictive validity? If we use a count to measure, would a rate perhaps be better? But, even if we believe our measurement is valid, there are still potential sources of error in the measurement process.

Sources of error An error in a measurement is a discrepancy between the measured value and the true value. “Error” does not mean mistake. Error is inevitable in any measurement.

Sources of error Two different sources of errors in measurements: 1.Bias 2.Variability

Sources of error Bias: comes from the way we measure. Systematically wrong in the same direction. Example Using police reports to measure public safety. Problem: not all crimes are reported to the police. Result: a biased measurement.

Sources of error How can we reduce bias? Bias comes from instruments or measuring procedures. Conclusion: the only way to reduce bias is to improve the instrument or use a better procedure.

Sources of error Variability: repeated measurements of the same individual give different values. Example Analog bathroom scale: each time you step on you may get a slightly different measurement of your weight.

Sources of error All measurements will have some variability. Example Measuring time with an atomic clock. Every 10 days the NIST reports the amount of error in time measurements. Some recent values: 6.0 ns, 5.9 ns, 5.5 ns, 5.2 ns Note that 1 ns = seconds

Sources of error If a measurement has small variability, then repeated measurements of the same individual will be close to each other. If a measurement has small variability, we say it is reliable.

Sources of error One way to reduce variability (make a measurement more reliable): Take several measurements, and then take the average. Variability causes measurements to be randomly scattered around the true value— some too small, some too large. Averaging several measurements will reduce the affect of this scattering.

Sources of error The two sources of error—bias and variability—are independent of each other. A measurement could have high bias but low variability, or vice versa. The following picture (which we have seen before) is helpful in understanding the two sources of error:

Sources of error Consider shooting arrows (measurements) at a target (the true value): Bias means the archer systematically misses in the same direction. Variability means that the arrows are scattered.

Sources of error Whenever we collect data, we make measurements. 1.Make sure measurements are valid (look for predictive validity, or use a rate instead of a count) 2.Measurement = true value + bias + variability 3.Averaging several measurements can reduce variability

Sample surveys are measurements A key example of a measurement is a sample survey. We attempt to measure a parameter using a statistic, but our measurement may have some bias, and will certainly have variability statistic = parameter + bias + variability

Bias, Variability, and Validity Keep in mind that bias and variability are both independent of validity. We can have a measurement that has very low bias and low variability, but if it isn’t valid, it’s useless. We can measure height very accurately, but that doesn’t make it a valid way to measure college readiness.