Case Study #9 Jackie Adams: Evaluating a Federally Funded Faculty Training Program Melody Murphy AIL 606 The University of Alabama

Part One: Creating an Evaluation Plan Jackie Adams has a instructional design position with a large university for a federally funded project. Her main responsibilities were to work with technical subject matter experts in the development, delivery, and evaluation of in-service faculty education. The name of the project Jackie was working on was the Advanced Manufacturing Technology Education (AMTE).

Part One: Continued The goal of the AMTE project was one of many projects funded by the Advanced Technology Education (ATE) program. Its goal was to improve science and engineering technician education at the undergraduate and secondary school levels. Also, it focused on curriculum development, instructional materials development, teacher/faculty enhancement, and/or student recruitment.

Main Goal Its main goal was to focus on teacher enhancement as a means of advancing technology education. The rationale behind AMTE was to provide educators with state-of-the-art knowledge in the technical disciplines, so that their students would in turn benefit from their teachings. Over a three-year grant, the AMTE project was to provide faculty development to 100 science, math, and engineering technology educators each year.

Progress Jackie had been working on the project for about four months when Ray DeMilo (her boss) asked her to develop an evaluation plan for the project. In order for them to receive funding for the project an evaluation plan was going to be required to assess the performance of their project.

Jackie’s Plan for the Evaluation Plan Give it to her boss. Send it to the funding agency. File it with the university grants office. Use it to conduct the evaluation.

Jackie’s Planning Evaluation Notes Why Evaluate? Evaluate What? How should the evaluation be conducted? Who should be involved? When should it be done?

Preliminary Analysis Questions for Part One – Question #1 Based on Jackie’s questions, what does she need to do to complete the evaluation. First of all Jackie decided that on why the program must be evaluated. It should be evaluated for the ongoing funding of the program and to measure whether or not learning is taking place in the workshops

Preliminary Analysis Questions for Part One – Question #2 How can Jackie determine whether short- and long- term outcomes are being achieved? What evaluation data does she need to collect? Short-term Jackie can conclude by receiving the surveys from the faculty about how they are using what they have learned in the workshops to conclude that by what they have learned they can use in their classrooms. Long-term Jackie can conclude from the surveys that if the faculty can continue or use the information now that they can use then the faculty can use it for long range planning in their classes.

Preliminary Analysis Questions for Part One – Question #3 How should Jackie design the instruments for this evaluation? Based on the material taught, she wants to know what the learners have gained from the workshops in order to report to the team that is reviewing her project in order to keep the grant money.

Part Two: Conducting a Meta-Evaluation

Conducting the Evaluation The funding agency sent out a six-member team to review the project’s progress and impact. The team would report the success of the project to key administrators at the university and the funding agency. The team included four post-secondary educators with a background in science, engineering, and technology and two engineering practitioners.

Team Review During the two-day meeting, Hank spent the entire first day with Jackie. He reviewed the progress and impact of the project. At the end of their meeting, Hank felt that confident that he had enough information to start the review.

Preliminary Analysis Questions Question #1 Part A: Questions Related to the Evaluation Plan What are the main evaluation questions Jackie appears to be addressing? Why – why evaluate the project? What – what to evaluate in the project? What should be measured? How – How should it be evaluated? Who – Who should be evaluated? When – When should it be evaluated? Periodically?

Preliminary Analysis Questions Question #2 What are the strengths and weaknesses of the evaluation design for answering the questions? Strengths – the evaluation plan answers why evaluate the objectives (reason for evaluation), purpose, when it should be done which is periodically. Weaknesses – How it should be done. It says documentation but it does not specify the documentation used or the learners in Jackie’s evaluation plan.

Preliminary Analysis Questions Question #3 Given the goals stated in the ATME proposal, are these questions sufficient? Yes, it does give the basic why, what, how, who, and when.

Preliminary Analysis Questions Question #1 Part B: Questions Related to Instrumentation How appropriate are the instruments for the goals of the evaluation? I question the validity. This is a lot of Jackie’s opinion on the evaluation plan. I would probably recommend incorporating interviews and surveys to the evaluation plan.

Preliminary Analysis Questions Question #2 Are the evaluation questions answered in enough detail, or are other instruments needed? I felt like there could have been more detail, for example, when stated “determine how disciplined and effective the organization’s operations are.” How will these be determined?

Preliminary Analysis Questions Question #3 What, if anything, would you change about the instruments and why? I would add a survey to be given to the faculty/staff. For reinforcement for what is stated in the evaluation plan.

The instructional design of high-tech areas using computers. What are the implications for instructional design practice in this case? The instructional design of high-tech areas using computers. For example computer numerical control, programmable logic controllers, robotics, electromechanical controls, lasers, solid modeling, and rapid prototyping in the workshops conducted by Jackie.

As an instructional designer, what did I learn? In order to keep your program or project funded it is important to regularly evaluate it. Ask questions such as: Why, what, how, who, and when.

My Questions How important do you think evaluation plans are to instructional design projects? What types of instruments would you recommend for Jackie to use in helping with the evaluation of her project?