Biases in Experimental Design: Validity, Reliability, and Other Issues

Slides:



Advertisements
Similar presentations
What is Public Opinion? Public opinion can be described as those attitudes held by a significant number of people on matters of government and politics.
Advertisements

Chapter 3 Producing Data 1. During most of this semester we go about statistics as if we already have data to work with. This is okay, but a little misleading.
Chapter 5: Descriptive Research Describe patterns of behavior, thoughts, and emotions among a group of individuals. Provide information about characteristics.
Introduction To Research 589(A)
Gile Sampling1 Sampling. Fundamental principles. Daniel Gile
Boundless Lecture Slides Free to share, print, make copies and changes. Get yours at Available on the Boundless Teaching Platform.
Research Design ED 592A Fall Research Concepts 1. Quantitative vs. Qualitative & Mixed Methods 2. Sampling 3. Instrumentation 4. Validity and Reliability.
Introduction Studies are important for gathering information. In this lesson, you will learn how to effectively design a study so that it yields reliable.
Module 2 Research Strategies. Scientific Method A method of learning about the world through the application of critical thinking and tools such as observation,
Designing psychological investigations
Copyright © 2009 Pearson Education, Inc. Chapter 13 Experiments and Observational Studies.
Chapter 2 Section 2 Surveys, Samples, and Populations Obj: Explain the survey method and the importance of proper sampling techniques.
Understanding Populations & Samples
Copyright © 2009 Pearson Education, Inc.
Sampling and Sampling Distribution
Introduction to Survey Research
Experimental Research
Business Research Methods William G. Zikmund
Chapter 2: The Research Enterprise in Psychology
Approaches to social research Lerum
Research Methods and Statistics
The Science of Social Psychology
Principles of Quantitative Research
Reliability and Validity
Sampling Distributions and Estimation
Classification of Research
Some General Concepts of Point Estimation
Non-Probability sampling methods
Bias and Error What can go wrong?.
Distribution of the Sample Means
The Experimental Method
Research Design: Terms to Know
Inferential Statistics and Probability a Holistic Approach
Chapter 5 Sampling Distributions
CHAPTER 12 Sample Surveys.
Inference for Sampling
Statistics: the language of psychological research
Chapter 5 Sampling Distributions
Principles of Experiment
Chapter 2: Psychological Methods
Module 02 Research Strategies.
By C. Kohn Waterford Agricultural Sciences
Welcome.
Module 8 Statistical Reasoning in Everyday Life
1.2 Sampling LEARNING GOAL
Chapter Two Research in Psychology
Unit 1 Lesson 4: Surveys, Experiments, and Observational Studies
Data Collection and Sampling
Basic Research Methods
Psychological Research Why do we have to learn this stuff?
Basic Concepts in Social Science Research
Sampling Distributions
Daniela Stan Raicu School of CTI, DePaul University
Experiments II: Validity and Design Considerations
Chapter 5: Producing Data
WHAT IS PUBLIC OPINION? DEFINITIONS
Mass Media and Public Opinion
An overview of the thesis project
Bias in Studies Section 1.4.
An overview of the thesis project
(-4)*(-7)= Agenda Bell Ringer Bell Ringer
Methodological Evaluation of Experiments
Statistics Definitions
What do Samples Tell Us Variability and Bias.
Chapter 8 SAMPLING and SAMPLING METHODS
Levine et al continued.
Randomization and Bias
MGS 3100 Business Analysis Regression Feb 18, 2016
Chapter 5: Sampling Distributions
Some General Concepts of Point Estimation
Presentation transcript:

Biases in Experimental Design: Validity, Reliability, and Other Issues

TAKE NOTE Research studies with small sample sizes, high variability, and sampling bias are usually not representative of the general population. Sampling bias is when the sample in question is not representative of the general population. Selection bias occurs when the participants in the sample are not equally and fairly selected for both the experimental and control groups; this renders any results from the experiment meaningless.

Take Note Response bias is when only highly motivated people return a survey. When this occurs, the resulting data is biased toward those with the motivation to answer and submit the survey, and is therefore not representative of the population as a whole. External validity is the ability to apply conclusions gathered from the results of an experiment to the general population. Data sets with little variability have values that are similar to each other; data sets with high variability have values that are more spread out.

Key Terms Reliability: overall consistency of a measure External Validity: whether or not study findings can be generalized to real world scenarios Bias: inclination, predisposition, or prejudice toward something Law of Diminishing Returns: the tendency for a continuing effort toward a particular goal to decline in efficacy after a certain amount of success has been achieved Key Terms

Overarching Ideas in Research Research studies often fall prey to experimental bias, in which the results are not representative of what they are supposed to measure. This limits the applicability of the results to anything beyond the experiment itself, which decreases or eliminates the value of those results.

External Validity A study that is externally valid is one in which the data and conclusions gathered from the results of an experiment can be applied to the general population outside of the experiment itself. If the study's data and conclusions cannot be applied to the general population, including general events or scenarios, then the experiment's results are only relevant to that experiment, and nothing more. A study's external validity can be threatened by such factors as small sample sizes, high variability, and sampling bias.

Small Sample Sizes The smaller the sample size for an experiment, the less applicable the results will be to the general population. The world has some 7 billion individuals, and thus a representative sample in any experiment would have to be very large to be applied to this general population. Nonetheless, the larger the sample group is in relation to the general population to whom the results are to be applied, the more likely it is to be applicable.

Small Sample Sizes This premise, however, can be negatively impacted by the law of diminishing returns, which states that effectiveness will decline after a certain amount of success has been achieved. This means that after a certain point, including more individuals in a study would gradually have less value to researchers. This could be caused by a multitude of factors, including cost and time put into the research. Generally it is best to attain a reasonable sample size that is representative of the population being studied.

High Variability Variability, also known as dispersion or spread, refers to how spread out a group of data is, or how much the measures differ from each other. Data sets with similar values are considered to have little variability because the values are within a smaller spread, whereas data sets with values that are spread out have high variability because the values are within a larger spread. In many instances of high variability there are outliers, which are values that exist far outside of the area where the majority of values are found. In many cases these outliers, which increase the variability of the data set, are removed when conducting statistical analysis of the data.

Sampling Bias Sampling bias occurs when the sample participating in the study is not representative of the general population. This may be the result of purposeful selection of participants by the researcher, but there are many other factors that can create sampling bias. One example is surveys taken during a presidential election. The results of the surveys often depend on the city, state, or area being surveyed. For example, people in cities tend to vote one way, while people in rural environments often vote another. Similarly, one's geographic location (the Northeast, South, Midwest, etc.) can have an impact on who is being surveyed. If there is a high saturation of a given political party in an area surveyed, then the results will be skewed in the direction of the political party, and not be representative of the general population.

Selection Bias Selection bias happens when the comparisons in data from the sample population have no meaning or value because the participants in the sample were not equally and fairly selected for both the experimental and control groups. Both the experimental and control groups should be representative of the general population, as well as representative of each other. One group should not show substantially higher characteristics of a given variable than the other, as this can distort the findings.

Response Bias Response bias (also known as "self-selection bias") occurs when only certain types of people respond to a survey or study. When this occurs, the resulting data is biased towards those with the motivation to answer and submit the survey or participate in the study. The resulting data, however, is not representative of the desired sample, nor the population at large. This is because only a select few have answered the survey and participated in the experiment. This data requires a disclaimer saying that out of all respondents, a certain characteristic is found. Regardless of a disclaimer, the results cannot be applied to the general population, nor the entire desired sample group.