Designing an Evaluation Plan

Get it rolling… To generate a good plan means logically working through a series of issues  stakeholders and their concerns  constraints  translate concerns into key evaluation questions  selection of data gathering methods to address key questions that are to be the focus

When do you make a plan? Planning for evaluation should occur as part of the other planning activities associated with project start up.

Form a team An evaluation group should be established and basic management issues need to be addressed

Identifying stakeholders Understanding the stakeholders and the audience of the evaluation report(s) will shape:  the goals/objectives of the evaluation  the questions to be asked and when  the methods of data collection, analysis and reporting

Identifying concerns Stakeholders will likely differ in their concerns and what they want to find out, but these are not necessary mutually exclusive. Concerns will vary from project to project. Concerns will change over the life of the project.

Stages Pre-implementation  Which concerns need to be addressed during the design and development of the project? Post-implementation  Short term  Medium term  Long term

Constraints? These factors will determine the size and scale of the evaluation and what the evaluation team can do practically.  Budget and resources  Time  Availability of competent staff  Pre-specified evaluation objectives, methodologies and/or reporting procedures  Legal or ethical issues  Availability of data  ‘Political’ considerations

The questions… You must spend time on getting the evaluation questions right.  OR, you may get the wrong answers, or answers to questions you didn't ask or want to know about. Action Questions High Value Questions

Data Gathering It is best to use a number of data gathering techniques and/or sources of data to substantiate findings.  This is known as a process of triangulation — the use of multiple investigative methods or information sources to get the answer to the question at hand.

Data Sources Students — prospective, current, past, withdrawn Colleagues — teaching partners, tutors, teachers external to the project Discipline/instructional design experts Professional development staff Graduates and employers Documents and records — teaching materials, assessment records, past SETLs, assessment statements and tasks

Selecting a methods… Paradigm for the study (empirical, interpretive, critical theory-based, pragmatic) Time involved in preparing to use the particular method/tool (e.g. preparation of a bank of questions for a questionnaire) Time involved in gathering or recording the data — on the part of the data collector; on the part of the 'evaluee/s' the time needed to analyse and report the data the scale involved — the number of students, staff required for valid/authentic data.

Method – Now what? The skill/expertise required to use the method The expertise, personnel and/or resources required to analyze and/or report the data.

Storing the data Making sure that data is safe and not lost Thinking through filing categories; e.g. by question type; data source; data method Considering confidentiality requirements & other safeguards arrangements to access data

Ethical Considerations In any evaluation, the rights and welfare of 'subjects' need to be respected and protected. Privacy: Some data gathering techniques may be perceived as an invasion of privacy if prior consent on the part of the subject(s) has not been gained. Confidentiality: Much information that subjects provide is given in confidence unless specific permission to use 'private' information.

Websites to explore