…an open, online course Statistics in Education for Mere Mortals Statistics in Evaluation & Research: Some Important Context Lloyd P. Rieber Professor of Learning, Design, & Technology The University of Georgia

Evaluation Research Consider the difference between evaluation and research

Research Evaluation Process Descriptive Judgmental Purpose Testing theory and producing generalizable findings Determine the effectiveness of a specific program or model Origin of Questions Scholars in a discipline Key stakeholders and primary intended users Who Judges Quality and Importance? Review by peers in a discipline Those who use the findings to take action and make decisions Ultimate Test of Value Contribution to knowledge Usefulness to improve effectiveness http://betterevaluation.org/blog/framing_the_difference_between_research_and_evaluation

Research Evaluation Research Evaluation Evaluation Research Evaluation

Instructional Congruency Objectives Instruction Evaluation

Characteristics of Good Assessment Instruments Validity Does the instrument assess what it is supposed to assess Reliability People who “know the material” do well, those who don’t do poorly; consistency Practicality The instrument can be implemented with relative ease Efficiency The instrument takes as little time as necessary to get valid and reliable results

Kirkpatrick’s Levels of Evaluation Level One Level Two Level Three Level Four Reaction Learning Behavior Results The purpose for evaluation is to determine the effectiveness of a training program. According to this model, evaluation should always begin with level one, and then, as time and budget allows, should move sequentially through levels two, three, and four. Information from each prior level serves as a base for the next level's evaluation.

Useful Data Quantitative Qualitative Mixed Methods

Qualitative Research Focus on phenomena that occur in natural settings. Involve studying those phenomena in all their complexity. The researcher is an instrument. Disciplines that conduct qualitative research: anthropology, sociology, history, political science, medicine, psychology, and education.

Qualitative Research Designs Five common qualitative research designs Case study Ethnography Phenomenological study Grounded theory study Content analysis

Sampling Sample: the particular entities researchers select Sampling: the process of selecting the sample Qualitative researchers are often purposeful in their selection of data sources.

Collecting Data from People https://www.flickr.com/photos/mmoorr/7750344104 https://www.flickr.com/photos/9110880@N04/7636476418 http://imgarcade.com/1/serious-black-business-woman/ http://pixabay.com/en/office-man-manager-businessman-207219/

Consider Mixed Methods Completeness Complementarity Hypothesis Generation and Testing Development of Appropriate Research Tools & Strategies Triangulation Resolution of Puzzling Findings

http://www.fortsaulsburyde.com/FC-Towers.html

What is “Good” Education Research? Karl Hostetler “As we do our work, we need to think beyond questions of how we will study students or analyze school policies: We need to think about how we can make life better for people. We need to think beyond our taken-for-granted ideas of well-being and what is good and make those ideas the objects of serious, communal inquiry. Serving people’s well-being is a great challenge, but it is also our greatest calling.” (p. 21) Hostetler, K. (2005). What Is "Good" Education Research? Educational Researcher, 34(6), 16-21.

…an open, online course Statistics in Education for Mere Mortals Statistics in Evaluation & Research: Some Important Context Lloyd P. Rieber Professor of Learning, Design, & Technology The University of Georgia