Chapter 16 Measuring in Research

Measurement Challenges in Research in PE, Sport, & Exercise Science For physical education:For physical education: –Involvement of many different variables –Influence of state and local education standards –Difficulty in qualifying and quantifying learning Difficulty in replicating activities or injuries in laboratory conditionsDifficulty in replicating activities or injuries in laboratory conditions Unknown elements and complex measurements in exercise physiologyUnknown elements and complex measurements in exercise physiology

Field vs. Laboratory Studies Field studies:Field studies: –Conducted in the actual context of the measurement rather than in an artificial setting –Advantages: Realistic; high ecological validity –Disadvantage: Loss of control over variables affecting the measurements Laboratory studies:Laboratory studies: –Simulations in a setting where the environment is controlled

Quantitative and Qualitative Research Qualitative research:Qualitative research: –Seeks to understand human behavior –Uses small but intensely studied samples Quantitative research:Quantitative research: –Uses experimental measures to test hypotheses; extrapolates results from a sample to a larger population –Uses random samples; unbiased representation of the population

Measurement Challenges in Quantitative Research Getting an overview of the data—the “big picture”Getting an overview of the data—the “big picture” Using non-normally distributed dataUsing non-normally distributed data Confusing practical vs. statistical differencesConfusing practical vs. statistical differences Choosing appropriate research subjectsChoosing appropriate research subjects Measuring the right variablesMeasuring the right variables(continued)

More Challenges Recognizing ceiling effects, floor effects, and regression to the meanRecognizing ceiling effects, floor effects, and regression to the mean Relying too much on group meansRelying too much on group means Using sample sizes that are too smallUsing sample sizes that are too small Making mistakes in data interpretationMaking mistakes in data interpretation Having equipment issuesHaving equipment issues

Ceiling Effects, Floor Effects, and Regression to the Mean Ceiling effect:Ceiling effect: –Research subjects are near the maximal score and cannot be expected to exceed the “ceiling” value. Floor effect:Floor effect: –Research subjects are near the lowest possible score and cannot be expected to go below the “floor” value. Regression to the mean:Regression to the mean: –The tendency for extreme scores to move toward the average (mean) the second time a test is taken.

Power Analysis of a Statistical Test The probability that the test will find a statistically significant difference between measures.The probability that the test will find a statistically significant difference between measures. Five factors:Five factors: –Effect size –Sample size –Alpha (p) level –Variability of the data –Tails (one- or two-sided t-tests) Online:Online:

Power Errors 1.Making a Type II error: Failing to find a difference between two groups/ treatments that really does exist. 2.Making a research decision based on a Type II error.

Your Viewpoint Can you think of an example of wrong conclusions being drawn, or of data being overinterpreted, in an experiment or research hypothesis?Can you think of an example of wrong conclusions being drawn, or of data being overinterpreted, in an experiment or research hypothesis? What happened? What was the result?What happened? What was the result?

Equipment Issues Equipment validity: How accurate is it?Equipment validity: How accurate is it? –Calibration: References the equipment to a known, valid measurement. Equipment reliability: Is it dependable?Equipment reliability: Is it dependable? Equipment objectivity:Equipment objectivity: –Is it being used in the same way by every person? –Are instructions clear? Are they being properly followed?

Evaluating Equipment for Purchase Establish specifications in terms of validity, reliability, and objectivity:Establish specifications in terms of validity, reliability, and objectivity: –How much accuracy do you need? –How reliable is it? –How difficult is it to use? Review the cost:Review the cost: –Initial cost –Cost of upkeep or operation costs

Challenges in Clinical and Epidemiological Quantitative Research Sensitivity: Probability of a positive test among patients with a disease.Sensitivity: Probability of a positive test among patients with a disease. Specificity: Probability of a negative test among people without a disease.Specificity: Probability of a negative test among people without a disease. Predictive value:Predictive value: –Proportion of patients correctly diagnosed who displayed positive test results. –Must be greater than the proportion of the disease in the population.

Measurement Challenges in Qualitative Research Establishing validity:Establishing validity: –Honesty and completeness of data collection –What did investigators do, how much, and how often? Establishing reliability:Establishing reliability: –Internal-consistency reliability; confirmability Establishing objectivity:Establishing objectivity: –Biases of investigators and methods of data collection –No research or measurement is ever free of subjectivity

Increasing Objectivity in Qualitative Research Have multiple investigators analyze the dataHave multiple investigators analyze the data Have computer programs analyze dataHave computer programs analyze data Use triangulation:Use triangulation: –Find agreement among three different perspectives or sources of data

Personal Disclosure Statement for a Qualitative Study

Your Viewpoint Think of the magazines and journals you read regularly. Do you think the authors’ backgrounds may have influenced their writings and opinions?Think of the magazines and journals you read regularly. Do you think the authors’ backgrounds may have influenced their writings and opinions? If so, how will this alter the way you read their articles?If so, how will this alter the way you read their articles?

Reading Research Papers Find relevant research papers.Find relevant research papers. Read quantitative journal articles.Read quantitative journal articles. Understand the peer-review evaluation process.Understand the peer-review evaluation process.

Elements of Quantitative Journal Articles

Some Issues Considered in the Peer-Review Process How relevant/original is the topic?How relevant/original is the topic? Are the measurement techniques appropriate?Are the measurement techniques appropriate? Are the study methods presented in enough detail?Are the study methods presented in enough detail? Is the analysis free from major error?Is the analysis free from major error? Are any conclusions not supported by the data?Are any conclusions not supported by the data? Does anything else explain the results?Does anything else explain the results? Did the authors overlook anything?Did the authors overlook anything? Does this manuscript contribute to the relevant body of knowledge?Does this manuscript contribute to the relevant body of knowledge?