Evaluation Research and Engineering Education Lesley Jolly For AaeE ERM wiki at

What is evaluation research? Periodic assessment of results –Appropriateness, effectiveness, efficiency, impact, sustainability Identifies intended and unintended effects Identifies what worked and what didn’t Provides level of judgement about overall worth of intervention

What has it got to teach us? Systematic approach to what we’re doing already Extracting lessons learned Legacy of sustained evaluation frameworks for ongoing data collection Arguments against ‘popularity contest’ course and program evaluation

Step 1: describe the program Aims and objectives –Need to be clearly articulated –Does everyone involved share the same aims Program logic diagram –Describes how we think the program produces results –May be called logframe

Program logic identifies what we need to evaluate

Evaluating PBL tutor training INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACTS Training materials Staff time Staff support workshops Rooms/ space Library support [no Administrative support] Published research Compiling materials Putting Materials online Running workshop Staff research Review and edit materials Staff reflection Numbers of new tutors Numbers of Returning tutors Level of satisfaction Growing library of examples and materials Tutors facilitate rather than inform Tutors Understand PBL Tutors develop Knowledge independently Tutors make PBL - Appropriate contributions to Curriculum review Students take Responsibility for learning Students learn to focus research and Synthesise results

Monitoring facilitates data collection Can be process (outputs) or impact (short to medium outcome) monitoring Need to develop indicators of progress Targets may be included in indicators or separate e.g. –In % of 2 nd yrs will have used the new facility for more than 20 hrs –OR percentage of students using facility (indicator) –50% in 2010 (target)

PBL Tutor training GOAL: provide timely well-placed supportive guidance to encourage tutors to scaffold and facilitate student learning OBJECTIVES: at the end of training the successful tutor will be able to articulate a good understanding of the objectives and methods of PBL guide student learning through providing appropriate support and guidance rather than information contribute to curriculum development within a staff team

Monitoring Tutor Training FOCUSPERFORMANCE INDICATORS DATA SOURCES DATA COLLECTION METHODS RESPONSIBILITY FOR COLLECTION TIME FRAME Objective 1: tutors articulate a good understanding of the objectives and methods of PBL Outcomes Tutors facilitate rather than inform Tutor contributions to discussion Online Discussion lists Content analysis of lists Course co-ordinatorTwice per semester Outputs Level of satisfaction Likert: training has clarified purposes of PBL Exit surveyPaper survey post training TrainerEvery semester Objective 2: tutors guide student learning through providing appropriate support and guidance rather than information Outcomes Tutors facilitate rather than inform Students take responsibility for learning Tutor contributions to discussion Student contributions to discussion Online Discussion lists Online Discussion lists Content analysis of lists Course co-ordinator Twice per semester Twice per semester

Evaluation asks formative and summative questions Are the questions cohesive and logical? Do evaluation questions link to monitoring data? Have ethical issues been addressed? What mechanisms are in place to gather the learnings generated by evaluation? What needs to be retained from this evaluation process in future years?

Vary questions to suit program QUESTIONSSOURCES OF INFORMATION FROM MONITORING SOURCES OF INFORMATION FROM EVALUATION DATA COLLECTION DATA ANALYSIS AND REPORTING METHODS Is the program as implemented appropriate, and is it being appropriately monitored? Has the purpose of the program been achieved? Is the program being implemented in the most efficient and cost effective way possible? Has the program produced sustainable results and outcomes?

PBL Tutor Training Appropriateness Did the training model the target behaviour? Has the purpose of the training been achieved? Indicator 1: changes in Tutor behaviour indicating deeper knowledge of and commitment to PBL Indicator 2: changes in student behaviour Efficiency Was the time invested by staff good value? Sustainability what needs to retained as core material from year to year to retain benefit of training

Owen, J. (2006) "Program Evaluation" Allen & Unwin: Crows Nest Making use of evaluation

Dissemination and reporting Findings may be communicated throughout project to multiple audiences –Evidence, conclusions, judgements, recommendations. Different occasions call for different styles –Oral, interactive workshops, posters, reports, summaries, papers, conference presentations Must be well timed

Developing capacity “process use of evaluation” –Taking part develops skills –Taking part sensitises staff to issues Improved communication –Ongoing 360 degree dissemination –Development of local discourse