1. Part II Sigma Freud & Descriptive Statistics
Chapter 6   
Just the Truth:
An Introduction to Understanding Reliability and Validity

2. What you will learn in Chapter 6
What reliability and validity are and why they are important
Basic measurement scales
Computing and interpreting reliability coefficients
Computing and interpreting validity coefficients

3. Why Measurement?
You need to know that the data you are collecting represents what it is you want to know about.
How do you know that the instrument you are using to collect data works every time (reliability) and measures what it is supposed to (validity)?

4. Scales of Measurement
Measurement is the assignment of values to outcomes following a set of rules
There are four types of measurement scales
Nominal
Ordinal
Interval
Ratio

5. Nominal Level of Measurement
Characteristics of an outcome that fits one and only one category
Mutually exclusive categories such as male or female, Caucasian or African-American, etc.
Categories cannot be ordered meaningfully
Least precise level of measurement

6. Ordinal Level of Measurement
Characteristics being measured are ordered
Rankings such as #1, #2, #3
You know that a higher rank is better, but not by how much

7. Interval Level of Measurement
Test or tool is based on an underlying continuum that allows you to talk about how much higher one score is than another
Intervals along the scale are equal to one another

8. Ratio Level of Measurement
Characterized by the presence of absolute zero on the scale
An absence of any of the trait being measured

9. Things to Remember
Any outcome can be assigned one of four scales of measurement
Scales of measures have an order
The “higher” up the scale of measurement, the more precise the data
More precise scales contain all of the qualities of the scales below it

10. Classical Test Theory: Os = Ts + E
Observed score
the actual score on a test
True score
theoretical reflection of the actual amount of a trait or characteristic an individual possesses
Error score
part of the score that is random
Reliability = True Score / True Score + Error

11. Types of Reliability Test-Retest Parallel Forms Internal Consistency
Measure of Stability
Parallel Forms
Measure of Equivalence
Internal Consistency
Measure of Consistency
Cronbach’s Alpha (coefficient alpha)
Inter-Rater
Measure of Agreement

12. Using the Computer
SPSS and Cronbach’s Alpha

13. How Big is Big? Reliability coefficients should be positive
0.0 to 1.0
General Rules of Thumb…
Test-Retest =
Inter-Rater = 85% agreement
Internal Consistency = .70 – 1.0
High Reliability DOES NOT mean quality!!

14. Establishing Reliability
Make sure instructions are standardized across all settings
Increase number of items or observations
Delete unclear items
Moderate easiness or difficulty of tests
Minimize the effect of external events

15. What is the Truth? Validity
The extent to which inferences made from a test are…
Appropriate
Meaningful
Useful
(American Psychological Association & the National Council on Measurement)
Does the test measure what it is supposed to measure?

16. Types of Validity
Traditionally speaking there are three types of validity evidence:
Content Validity
Criterion Validity
Predictive Criterion validity
Concurrent Criterion validity
Construct Validity

17. Content Validity
Property of a test such that the test items sample the universe of items for which the test is designed.
How to Establish…
Content Expert
Do items represent all possible items?
How well do the number of items reflect what was taught?

18. Criterion Validity
Assesses whether a test reflects a set of abilities in a current (concurrent) or future (predictive) setting as measured by some other test.
Concurrent Validity
How well does my test correlate with the outcomes of a similar test right now?
Predictive Validity
How well does my test predict performance on a similar measure in the future?

19. Construct Validity
Most interesting…most difficult source of validity to establish
Construct = group of interrelated variables such as...
Aggression
Hope
Intelligence
Want your construct to correlate with related behaviors and not correlate with behaviors that are not related.

20. All About Validity

21. Validity & Reliability
The “Kissing Cousins”
A test can be reliable but not valid
A test cannot be valid unless it is reliable because…
“A test cannot do what it is supposed to do (validity) until it does what it is supposed to do consistently (reliability).”

22. Part III Taking Chances for Fun and Profit
Chapter 7   
Hypotheticals and You: Testing Your Questions

23. What you will learn in Chapter 7
The difference between samples and populations
The importance of…
The null hypothesis
The research hypotheses
How to judge a good hypothesis

24. What is a hypothesis? An “educated guess”
Their role is to reflect the general problem statement or question that is driving the research
Translates the problem or research question into a form that can be tested.

25. Samples and Populations
The large group to which you would like to generalize your findings
Sample
The smaller, representative group of the population that is used to do the research
Sampling error – a measure of how well a sample represents the population

26. The Null Hypothesis
Statements that contain two or more things that are equal (unrelated) to one another
H0 : m1 = m2
The starting point and is accepted as true without knowing more information
Benchmark to compare actual outcomes

27. The Research Hypothesis
Statement that there is a relationship between two variables
Two Types…
Nondirectional -- H1 : X1 ≠ X2
Reflects a difference; direction is not specified
Two-tailed test
Directional -- H1 : X1 > X2
Reflects a difference; direction is specified
One-tailed test

28. Null & Research Hypotheses

29. Differences Between Null and Research Hypotheses
No relationship between variables
Relationship between variables
Refers to the population
Refers to the sample
Indirectly tested
Directly tested
Written using Greek symbols
Written using Roman symbols
Implied hypothesis
Explicit hypothesis

30. What Makes a Good Hypothesis?
Stated in a declarative form rather than a question
Defines an expected relationship between variables
Reflects the theory or literature on which they are based
Brief and to the point
Testable – includes variables that can be measured

31. Glossary Terms to Know Hypothesis
Null Hypothesis
Research Hypothesis
Directional & Non-directional hypotheses
One-tailed & Two-tailed test
Population
Sample
Sampling error