Program Evaluation Making sure instruction works.

Designers Mark Bullard Carmela DeRoy Ashley McCormick Lucia Salters Susan Trejo

Overview of Proposed Evaluation Plan This is a new version of the EMT training course. The plan is to use a variety of evaluation tools to ensure that the new course materials are congruent with the goals and objectives set forth by the client, to ensure that the materials and technology are congruent and effective, and to ensure that the new course offers a positive ROI for the client.

Design reviews At this stage of evaluation the design will be compared to the stated goals set forth by the client. The functionality, congruence, strategies for pace, relevancy, gaining and maintaining attention, tools and technologies, media will all be tested. Goal Review: Does the instruction meet the needs assessment? Task Analysis Review: Are prerequisite skills as indicated in the task analysis clearly stated? Are the learners accurately portrayed and are the lessons correctly designed for the target audience? Review of Assessments: Do the assessments and benchmarks reflect the course objectives precisely? Will the assessments clearly show which students are learning the material and which are not?

Small Group Review At this stage of evaluation the design will be used by a focus group of students to test for understandability, clarity, congruence, and to see if a sample group of representative students have the requires prerequisite skills to succeed in the class. The focus group will fill out a short questionnaire that utilizes the Likert scale to rate aspects of the course. There will also be a short discussion between the designers and the focus group that will go more into depth regarding the scaling of the course on the questionnaire. Revisions will be made according to feedback from the students. Was the technology easily understood and usable? Was the information presented above your pre-knowledge level? Was enough time spent on each section? Was too much time spent on any section? Were the objectives clearly stated and an obvious part of the instruction? Were the assessments clear and did they correlate to the stated objectives? Were there any erroneous or confusing questions?

Expert Review Before the course goes live an expert in the field of Emergency Medical Technicians will be brought in to rate the course. Again a Likert Scale will be used with space for additional comments. An instructional design review will be done where the expert will be asked to rate the multi media and instructional strategies and pace of the course. A visual design review will be done where the expert will rate the visual aspect of the materials to ensure that all graphics are correct, pleasing and relevant. Finally a functionality review will be done where the expert will rate the functionality of the learning course.

Level One Evaluation Level one evaluations are surveys that are given to students at the end of a course. Since this is a new course, facilitators need to know if the class is effective. Students are I the best position to critique the course. A Likert Scale will be used to quiz the students and there will be comment spaces at the end of each section for students to elaborate on their answers. Level one evaluations can be used to make minor changes to the course. The evaluations will be anonymous to ensure candid answers. Facilitators can also utilize the level one evaluation to suggest changes in the course. The client will use the level one evaluations to give insight into what areas need improvement. Benchmarking will be utilized with any score falling below 3 on a 4 point scale considered for change. Reasonable changes can be made during this phase. Major changes cannot be made to the instructional strategy based on the Level One Evaluation

Level Three Evaluation Six months after the completion of the course, students will be assessed to ensure knowledge retention. The results of this level 3 evaluation will be used to determine if the training course better prepares EMTs for the field and will be able to determine if the new course impacted the participants in a positive way. This will give the client insight as to whether they should continue to offer the course as it is or opt to overhaul the course and make major changes in delivery methods. The information used as comparison for knowledge retention will come from the level one evaluations and the assessment data collected during the class. The students will be observed on the job and a checklist will be employed to log the performance data. The knowledge level before and during training will be compared to the knowledge level six months after training.

Level Four Evaluations Level four evaluations will determine if the cost of training is offset by the outcome of training. The level four evaluation will take place at least a year after initial training and will coincide with yearly performance reviews. The information provided by a level four evaluation will show if the training adequately prepared participants to become EMTs. If students taking the new course perform better than students who took the previously offered course, officials can be confidant that the new course provides a positive ROI and the new course will continue to be offered. The information used to compare the participant’s performance will be gathered during initial training by observation and assessment scores. A control group will be used to compare performance. The control group will be a group that did not participate in the new training. All information will be gathered through normal testing administered by the Emergency Services Department. Students who have taken both the new training and the old training will be tracked for the first year to provide information for comparison.

Contextual Factors That Affect Results Level one evaluation: Implementation factors can affect the outcome of the evaluation. If not given enough time participants will rush through the evaluation and not give thoughtful answers. Level Three evaluations: If there is a time lapse between completion of the course and when the participants are instated as working EMTs, there may be a loss of information. Level four evaluations: The length of time between the completion of the course and the level four evaluation may pose a problem. Management may want faster answers to their questions about the success of the course and the ROI. Level four evaluations are cost prohibitive. It is difficult to separate outside variables that affect the performance of the participants and the control group on any given day.