MAINTAINING QUALITY IN BLENDED LEARNING: FROM CLASSROOM ASSESSMENT TO IMPACT EVALUATION PART I: DESIGNING AND EVALUATING LEARNING Suzanne Weinstein, Ph.D. Director of Instructional Consulting, Assessment, and Research The Schreyer Institute for Teaching Excellence The Pennsylvania State University

Part I Session Objectives As a result of this workshop, participants will be able to…  …write a student-centered, measureable learning objective  …map learning objectives to appropriate assessments  …use course-level evaluation data for program assessment

But first, some definitions

Assessment “Assessment is the ongoing process of establishing  clear, measurable expected outcomes of student learning, Suskie, L. (2009) Assessing student learning: A common sense guide. San Francisco: Jossey-Bass.

Assessment “Assessment is the ongoing process of establishing  clear, measurable expected outcomes of student learning,  ensuring that students have sufficient opportunities to achieve those outcomes, Suskie, L. (2009) Assessing student learning: A common sense guide. San Francisco: Jossey-Bass.

Assessment “Assessment is the ongoing process of establishing  clear, measurable expected outcomes of student learning,  ensuring that students have sufficient opportunities to achieve those outcomes,  systematically gathering, analyzing, and interpreting evidence to determine how well students learning matches the expectations and Suskie, L. (2009) Assessing student learning: A common sense guide. San Francisco: Jossey-Bass.

Assessment “Assessment is the ongoing process of establishing  clear, measurable expected outcomes of student learning,  ensuring that students have sufficient opportunities to achieve those outcomes,  systematically gathering, analyzing, and interpreting evidence to determine how well students learning matches the expectations and  using the resulting information to understand and IMPROVE student learning.” Suskie, L. (2009) Assessing student learning: A common sense guide. San Francisco: Jossey-Bass.

Evaluation “Evaluation is the analysis and use of data by faculty to make judgments about student performance. Evaluation includes the determination of a grade or a decision regarding pass/fail for an individual assignment or for a course.” Goldman and Zakel (2009)

A blended or hybrid course is one in which some face-to-face class periods are replaced with online activities +

Designing learning in a blended course

Designing learning in a (blended) course 1. Learning Objective 2. Assignment 3. Learning Activities Wiggins and McTighe (1998), Understanding by Design

Designing learning in a (blended) course 1. Learning Objective 2. Assignment 3. Learning Activities Face-to -face or Online? Face-to -face or Online?

Designing learning in a (blended) course 1. Learning Objective 2. Assignment 3. Learning Activities Face-to -face or Online? Face-to -face or Online? Integrate face-to-face and online elements

Designing learning in a (blended) course Learning Objective At the end of this course, students will be able to evaluate an empirical research article Assignment Team project: Find an empirical article on a topic of personal interest and collaboratively evaluate the article using a wiki Learning Activities a. Students will read an article b. Instructor will model an article critique in class

Learning Objective Write a specific, learner-centered, measureable objective Taxonomy Match the objective to the closest equivalent of Bloom’s taxonomy Evaluation How you will evaluate the students on this objective? What is the best format for the evaluation? Teaching Strategies What teaching activity you will use to prepare students to meet objective? What is the best format for the teaching strategies? Face-to-face/Online connection How are the face-to- face activities connected to the online activities? At the end of this course, students will be able to evaluate an empirical research article. EvaluatingTeam project: Find an empirical article on a topic of interest and collaborate to evaluate the article using a wiki. Instructor will model an article critique (in class). Instructor modeling the critique links to students practicing the critique in the online environment. Adapted from Synthesis Worksheet:

Creating Learning Objectives

Identify what the learner should know or be able to do by the end of the course A learning objective should…  …focus on the learner  …be observable (i.e. measurable)

Adapted from Eileen Herteis, The Gwenna Moss Teaching & Learning Centre Use Bloom’s taxonomy (1956) to develop learning objectives at various levels of learning Evaluation Synthesis Analysis Application Comprehension Knowledge

Adapted from Eileen Herteis, The Gwenna Moss Teaching & Learning Centre If you prefer, you can also use Krathwohl and Anderson version of the taxonomy Creating Evaluating Analyzing Applying Understanding Remembering

Chemistry Learning Objectives Students will be able to:  Identify the chemical deficiencies of the soil for a crop.  Distinguish between different types of chemicals based on the chemical deficiencies of the soil for a crop.  Develop a solution for the soil deficiencies in a proposal geared towards a business audience.

Evaluate the following learning objective 1. In this course you will understand the 3 major communication disorders.

Evaluate the following learning objective 2. In this course we will cover the religion of ancient Egypt.

Mapping Objectives to Assessment Strategies (mere examples) TraditionalElectronic Remember Multiple choice or matching questions Online quiz or fact-based game Understand Multiple choice or Matching questions Essay summary Online quiz or fact-based game Flow chart Concept map ApplyMultiple choice questions Calculate the answer to a problem Demonstrate a procedure Online quiz Do an online search Make a podcast that examines a theory

Mapping Objectives to Assessment Strategies (mere examples) TraditionalElectronic Analyze Distinguish between 2 theories Analyze data Reverse engineer a device Conduct and report on a web survey Collect and analyze media clips Contrast 2 blog posts Evaluate Evaluate an writing sample Appraise a research article Judge a marketing plan Moderate a discussion forum Evaluate a video or debate Peer evaluation CreateGenerate an expository essay Formulate a research plan Produce and deliver a presentation Generate an e- Portfolio Create a video Build a website

One last, but not least, important consideration: Rubrics Rubrics for digital assignments

Electronic assessment examples History course objective Students will be able to create personal histories that link to specific historical times. Family e-Album

Electronic assessment examples Information Science and Technology Students will be able to collaboratively develop summaries of research related to online games and virtual worlds. IST 440 Research Project

Give it a try Go to the following link. It will take you to a google form. The questions will walk you through the process of creating the appropriate assignment for a particular objective. =dDZSbS1XOHI4ZngtMzZLNno4ZkVnWHc6MQ

Let’s see the results  Blended Learning Workshop Activity - Spreadsheet Blended Learning Workshop Activity - Spreadsheet

Use course-level assignments to measure program goals Psychology Program Goal Students will be able to understand and interpret psychological data, graphs and statistics. Course Objective Students will be able to correctly interpret a correlation coefficient.

Performance by student

Performance by learning objective

Resources  Hybrid Courses, University of Wisconsin at Milwaukee:  Schreyer Institute for Teaching Excellence, Penn State:  Bloom’s Digital Taxonomy:  Bloom’s Digital Taxonomy- Rubrics  Evaluating Media Characteristics

CONTACT INFORMATION Suzanne Weinstein