How do we know it works? Evaluating Learning Technology Projects EDUCAUSE Learning Initiative Seminar Clare van den Blink

Seminar Goals Seminar participants will be able to…. Select project goals that can be evaluated. Identify relevant indicators to evaluate project goals. Identify data collection methods Assess considerations for developing evaluation activities Create an evaluation plan for an academic technology project that is tied to their project assumptions and strategies.

Please indicate the type of projects you’re interested in evaluating.

What are the key challenges in fully evaluating your projects?

A Framework Project Goals Focus of Evaluation Evaluation Design Overview Indicators - measures of ”success” Data collection -Methods -Population -Procedures Timeline Data Analysis Reporting Findings

Introduction How this process was developed… “To evaluate the effectiveness of the technology enhancement and it’s impact on student learning” …..With constraints of staff, time and budget limitations.

Staff Effort Complexity of Evaluation Low High Small Project with technology intervention LMS Pilot Project- Service Decisions Quasi-experimental Research on Tech Intervention Importance-Complexity

Staff Effort Complexity of Evaluation Low High Small Project with technology intervention LMS Pilot Project- Service Decisions Quasi-experimental Research on Tech Intervention Importance-Complexity Interview Survey

Staff Effort Complexity of Evaluation Low High Small Project with technology intervention LMS Pilot Project- Service Decisions Quasi-experimental Research on Tech Intervention Importance-Complexity Interview Survey Interviews Surveys Tech Review Usability

Staff Effort Complexity of Evaluation Low High Small Project with technology intervention LMS Pilot Project- Service Decisions Quasi-experimental Research on Tech Intervention Importance-Complexity Interview Survey Interviews Surveys Tech Review Usability Interviews Surveys Observations Control group

To inform the evaluation Based on Prior research? What led to the development of the project? Literature review- What will inform the evaluation? Assumptions about the strategies & technologies selected?

Selecting Goals…. Since not all projects may be evaluated within the timeframe of the project… OR may be difficult to measure…

Selecting Goals…. Since not all projects may be evaluated within the timeframe of the project… OR may be difficult to measure… How can you SELECT goals that can be evaluated within the scope of the project?

Selecting Goals…. Since not all projects may be evaluated within the timeframe of the project… OR may be difficult to measure… How can you SELECT goals that can be evaluated within the scope of the project?  How would you PRIORITIZE the goals that are the most critical to evaluate?

Sample Goals: Review examples ofgoals and idenitfy goals that could be evaluated within the project constraints. EXAMPLE #1 Instructional Goals: 1.) Encourage active participation and critical thinking skills by using video clips of main stream movies to initiate class discussions. 2.) Encourage student involvement and active learning by creating a mechanism for students to record interviews in the field. (part of a class assignment) 3.) Create a repository of student-collected audio interviews for ongoing use in the curriculum. Audio clips will be used to illustrate the diversity of public education experiences. Other Project Goals:. 4.) Develop work flow and documentation for student recording of audio interviews and video clip processing. 5.) Choose, create, and provide an archiving mechanism for cataloguing clips.

Sample Goals: Review examples ofgoals and idenitfy goals that can be evaluated within the project constraints. EXAMPLE #2 Instructional Goals: A. ) Students will be able to practice application of fluid therapy, under various conditions, employing a unique computer-based simulation B.) Students will be able to interpret symptoms presented in a sick dog, select an appropriate treatment, administer fluids, monitor patient’s reaction and modify treatment plan accordingly. C.) Case simulation will enable students to experience clinical variability in a manner similar to hands-on practice. Other Project Goals: D.) Simplify the creation of a set of teaching models, or prototypes, that are the basis of the cases. E.) Create a method for generating unique computer- based cases that build from the prototypes. F.) Provide a method for saving case data for comparison.

Developing the evaluation plan

Evaluation Process Identify Project Goals Survey Observation Interviews Focus Groups Data analysis Case Study

Evaluation Process Identify Project Goals Focus of the Evaluation? What goals will be evaluated? Survey Observation Interviews Focus Groups Data analysis Case Study

Evaluation Process Identify Project Goals Focus of the Evaluation? What goals will be evaluated? Indicators: type of data to collect? Survey Observation Interviews Focus Groups Data analysis Case Study

Evaluation Process Identify Project Goals Focus of the Evaluation? What goals will be evaluated? Indicators: type of data to collect? Methods: how will data be collected? Survey Observation Interviews Focus Groups Data analysis Case Study

Evaluation Process Identify Project Goals Focus of the Evaluation? What goals will be evaluated? Indicators: type of data to collect? Methods: how will data be collected? Survey Observation Interviews Focus Groups Data analysis Case Study Population

Process: Select Goals …Focus  What goals will be the Focus of the evaluation? Project Goals 1.Use Personal Response System (polling) with questions to encourage critical thinking & student engagement. 2.Use of PowerPoint presentations with interactive lecture. 3.Implement use of a Tablet PC for annotating presentations for visual and interactive lectures Evaluation Process

Process: Select Goals …Focus  What goals will be the Focus of the evaluation? What is feasible to evaluate in project’s timeframe? Project Goals 1.Use Personal Response System (polling) with questions to encourage critical thinking & student engagement. 2.Use of PowerPoint presentations with interactive lecture. 3.Implement use of a Tablet PC for annotating presentations for visual and interactive lectures Focus of the Evaluation Use Personal Response System (polling) with questions to encourage critical thinking & active participation. (student engagement) Evaluation Process

Process: Select Goals …Focus  What goals will be the Focus of the evaluation? What is feasible to evaluate in project’s timeframe?  Identify what INDICATORS can be used to collect data, both indirect and direct measures. Project Goals 1.Use Personal Response System (polling) with questions to encourage critical thinking & student engagement. 2.Use of PowerPoint presentations with interactive lecture. 3.Implement use of a Tablet PC for annotating presentations for visual and interactive lectures Focus of the Evaluation Use Personal Response System (polling) with questions to encourage critical thinking & active participation. (student engagement) Evaluation Process

Evaluation Focus: Example 1 Formative: Since the project is assisting with the development of online modules, a formative evaluation of the modules will be conducted to look at the interface design, navigation, usability, organization and presentation of content, and the usefulness for student learning. The key focus will be on the “functionality” of the module.  Interface / Navigation / Design (not important in phase I)  Technology performance: test across browsers, OS, distance, etc.  Organization/presentation of content  Use of images, illustrations, images  Learning objectives** Summative: (part of the overall program evaluation) Since the project is assisting with the development of web-based modules, the summative evaluation will examine the effect on student perception of learning from the implementation of instructional technology in the course. Measures of success may include the student perception of greater ease in learning difficult concepts, and positive feedback about new modules. Evaluation Process

Select Methods Develop data collection METHODS for the indicators, such as surveys, interviews, observations, etc. What method(s) would you select evaluate the focus area- “functionality” of the online module? 1. Surveys 2. Interviews 3. Observation 4. Log analysis 5. All of the above 6. Other – post in chat Evaluation Process

Select Methods Develop data collection METHODS for the indicators, such as surveys, interviews, observations, etc What method(s) would you select to evaluate how well the online module met the “ learning objectives ”? 1. Surveys 2. Interviews 3. Observation 4. Log analysis 5. All of the above 6. Other – post in chat Evaluation Process

Other considerations  Identify the population from where the data will be collected. Can you contact this group?  Identity other data sources, such as logs, document, etc.  Are there considerations for human subjects research & informed consent on your campus? For example, using grades as data, what permissions are necessary? Evaluation Process

Timelines: How much time do you have? Need? Dec-Jan Project Transition & Closeout Fall Project Development Aug – Dec Fall Semester Project Implementation Implement Project Evaluation Project Evaluation Planning Evaluation: Data analysis & reports Evaluation Process

Evaluation Timeline Develop an initial timeline & staffing effort. Develop evaluation plan.September 1 Create an observation protocolSeptember 15 Observe EDU 271 during two class sessionsSept - October Complete the student survey. (instrument)October 1, 2005 Create an interview protocolOctober 15 Administer student survey.Mid- November Conduct student interviewsEnd of November Conduct data analysisDecember - January Complete evaluation reportFebruary 1 Evaluation Process

Evaluation Timeline Develop an initial timeline & staffing effort. Develop evaluation plan.September 1 Create an observation protocolSeptember 15 Observe EDU 271 during two class sessionsSept - October Complete the student survey. (instrument)October 1, 2005 Create an interview protocolOctober 15 Administer student survey.Mid- November Conduct student interviewsEnd of November Conduct data analysisDecember - January Complete evaluation reportFebruary 1 Where reality meets ideal evaluation methods…. Evaluation Process

Implementing the Plan & Reporting Results

Implementing Methods Surveys:  Identify or develop questions.  Do survey questions map to indicators?  Survey distribution & associated permissions. Interviews:  Develop interview questions & protocols.  Schedule and conduct interviews. Resources about quantitative and qualitative methods can guide development and implementation of methods and data analysis. Evaluation Process

Analysis & Reporting DATA ANAYSIS What type of analysis will be completed? Quantitative: survey analysis Qualitative: interview analysis based on interview protocols. Example: Overall, I am satisfied with the use of instructional technology in this course. Mean = Strongly Agree 82% 2 Agree 18% 3 Neutral 0 4 Disagree 0 5 Strongly Disagree0 Evaluation Process “I interviewed Prof. X about her experience with the simulation…..”

Analysis & Reporting Evaluation Process

Constraints

Staffing How can this planning all be completed within limited staff hours, while maintaining the INTEGRITY of the evaluation process? Evaluation Considerations & Tools

Do you think it is feasible to re-train staff for evaluation?

Staffing How can this planning all be completed within limited staff hours, while maintaining the INTEGRITY of the evaluation process?  How can staff be trained in this process without having a deep evaluation background?  What staff skills might be adapted?  Other campus resources? Evaluation Considerations & Tools

What type of existing skills could be adapted for evaluation?

Supporting Tools  Have an overall evaluation plan template that an be adapted to other projects.  Informed consent templates  Use common interview/observation protocols.  Develop question banks for survey questions.  Use of Video in Course Presentations and lectures  Use of Online instructional tutorials  Use of Presentations in Lecture Evaluation Considerations & Tools

question banks Evaluation Considerations & Tools

question banks Evaluation Considerations & Tools

Summary

Consider…  How can this METHODOLOGY be applied to your projects and institution?  How VIABLE is this as an evaluation methodology for your projects? When does a more ROBUST process need to be put in place?

A Framework Project Goals Focus of Evaluation Evaluation Design Overview Indicators - measures of ”success” Data collection -Methods -Population -Procedures Timeline Data Analysis Reporting Findings

A Framework 1. Project Goals 2. Focus of Evaluation 3. Evaluation Design  Overview  Indicators - measures of ”success”  Data collection - Methods - Population - Procedures 4. Timeline: what, when, how 5. Data Analysis 6. Reporting Findings

Questions?