1. …an open, online course
Statistics in Education for Mere Mortals Statistics in Evaluation & Research: Some Important Context
Lloyd P. Rieber
Professor of Learning, Design, & Technology
The University of Georgia

2. Evaluation
Research
Consider the difference between evaluation and research

3. Research Evaluation Process Descriptive Judgmental Purpose
Testing theory and producing generalizable findings
Determine the effectiveness of a specific program or model
Origin of Questions
Scholars in a discipline
Key stakeholders and primary intended users
Who Judges Quality and Importance?
Review by peers in a discipline
Those who use the findings to take action and make decisions
Ultimate Test of Value
Contribution to knowledge
Usefulness to improve effectiveness

4. Research Evaluation Research Evaluation Evaluation Research Evaluation

5. Instructional Congruency
Objectives
Instruction
Evaluation

6. Characteristics of Good Assessment Instruments
Validity
Does the instrument assess what it is supposed to assess
Reliability
People who “know the material” do well, those who don’t do poorly; consistency
Practicality
The instrument can be implemented with relative ease
Efficiency
The instrument takes as little time as necessary to get valid and reliable results

7. Kirkpatrick’s Levels of Evaluation
Level One
Level Two
Level Three
Level Four
Reaction
Learning
Behavior
Results
The purpose for evaluation is to determine the effectiveness of a training program. According to this model, evaluation should always begin with level one, and then, as time and budget allows, should move sequentially through levels two, three, and four. Information from each prior level serves as a base for the next level's evaluation.

8. Useful Data
Quantitative
Qualitative
Mixed
Methods

9. Qualitative Research
Focus on phenomena that occur in natural settings.
Involve studying those phenomena in all their complexity.
The researcher is an instrument.
Disciplines that conduct qualitative research: anthropology, sociology, history, political science, medicine, psychology, and education.

10. Qualitative Research Designs
Five common qualitative research designs
Case study
Ethnography
Phenomenological study
Grounded theory study
Content analysis

11. Sampling
Sample: the particular entities researchers select
Sampling: the process of selecting the sample
Qualitative researchers are often purposeful in their selection of data sources.

14. Collecting Data from People

15. Consider Mixed Methods
Completeness
Complementarity
Hypothesis Generation and Testing
Development of Appropriate Research Tools & Strategies
Triangulation
Resolution of Puzzling Findings

19. What is “Good” Education Research? Karl Hostetler
“As we do our work, we need to think beyond questions of how we will study students or analyze school policies: We need to think about how we can make life better for people. We need to think beyond our taken-for-granted ideas of well-being and what is good and make those ideas the objects of serious, communal inquiry. Serving people’s well-being is a great challenge, but it is also our greatest calling.” (p. 21)
Hostetler, K. (2005). What Is "Good" Education Research? Educational Researcher, 34(6),

20. …an open, online course
Statistics in Education for Mere Mortals Statistics in Evaluation & Research: Some Important Context
Lloyd P. Rieber
Professor of Learning, Design, & Technology
The University of Georgia