Use of an objective assessment tool to evaluate students basic electrical engineering skills Nandini Alinier University of Hertfordshire, U.K. Engineering Education Conference 2006

Overview Background & origins Application in an Engineering context Stages of OSTE –Development of stations –Preparation for sessions –Pilot –Implementation My perspective Students perspective Reflections Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Background Objective Structured Clinical Examination (OSCE) (Harden 1975) Originally developed in the mid-seventies in Dundee, Scotland –Used in medical education –Method to objectively assess the skills of medical trainees Used in a project funded by the British Heart Foundation at UH during (Alinier et al 2004) –Involved as an assessor –Lead investigators involvement Perceived positively by students & lecturers (Alinier 2003) Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

OSCE Comprises 15 to 20 stations Station: short task performed in 5 minutes –Practical exercise Requires an assessor –Theoretical exercise Pen & paper Gap period between stations (1 minute) –Students rotate individually through the stations –Allows time for assessors to complete marking & reset any equipment Timing of session by an electronic clock Students given explicit written instructions Objective marking sheet for each station Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Applied to Engineering Project undertaken in the School of Electronic, Communication & Electrical Engineering at the University of Hertfordshire, UK Funded by the Higher Education Academy Engineering Subject Centre (Learning & Teaching Support Network (LTSN) Engineering) Academic year Objective Structured Technical Examination (OSTE) Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Study about the OSTE Used for formative assessment Piloted with 2 nd year BEng students Conducted with 1 st year BEng students Circulated feedback questionnaires for students + assessors Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Stages of the OSTE Nandini Alinier, EE2006, 25 July, Liverpool, U.K. Stage No.Stage 1Development of OSTE stations 2Preparation for OSTE sessions 3Pilot of OSTE 4Running of OSTE

Stage 1: Development of OSTE stations Gathering a variety of ideas about essential electrical engineering skills –Seminar within School –Call for ideas via Identified appropriate range of exercises doable in 5 minutes Based on learning objectives for relevant group of students 16 stations were short-listed Developed instruction sheets & marking criteria for each station Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Stage 1: Development of OSTE stations (continued) Types of stations: –Theoretical: pen & paper –Practical: assessor required –Computer-based: assessor required Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Stage 2: Preparation for OSTE sessions Development of PC-based clock with an integrated voice prompt Briefing of students Training of staff to act as assessors Designing of student and assessor feedback questionnaires Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Stage 3: Pilot of OSTE Group of 15 volunteer 2 nd year BEng students Resulted in changes in: –Tasks to be carried out –Marking sheet –Instruction sheet Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Example of a computer-based station Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Stage 4: Running of OSTE 32 BEng students in the second semester of their 1 st year 16-station OSTE 2 sessions on the same afternoon Formative OSTE –Feedback given to students by assessor during the gap period Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

My perspective: benefits Identifying students strengths and weaknesses Students forced to do practical work individually Being assessment-driven, hence OSTE encourages students to participate Feedback given individually to students Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

My perspective: concerns Development of the OSTE stations is time consuming –Once the stations have been developed, running the OSTE session is fairly straightforward Running the OSTE is human resource intensive –However no need for marking after the session as most of it is done during the session A few students may find the OSTE stressful due to the intense assessment Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Students perspective Lets students know what is expected of them and what their strengths and weaknesses are –The OSTE session helped clarify what I was supposed to be able to achieve –I know my weaknesses and what I need to know Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Students perspective (continued) Students feel they have learnt something during the OSTE –as a result of immediate feedback from the assessors –OSTEs had addressed areas that would not have been picked up by other forms of lab work –when marking coursework and lab books, the lecturer might assume competence (e.g. in setting up an oscilloscope), when in fact the student had only just coped or had relied on their partner to do that part of the experiment –OSTE would show whether students could do the task: Effectively – getting the best settings, not just something usable Efficiently – because of the tight time limit Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Students perspective (continued) Unlike my concern, students did not find the exercise stressful Some people get a bit stressed before exams, whereas this is informal, you dont get so worried For similar reasons, students were against the idea of summative OSTEs, when they would probably attempt to cram specifically for the exam if were assessed for it were all going to prepare for it… rather than be tested on what we know Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Benefits Assesses specific key competencies that may be missed by other broader assessments Your lab skills are not always assessed, they just check your logbook and see your final result. This OSTE, it checks your method… how youre connecting equipment … not everyone uses [the equipment] properly in the right way Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Benefits (continued) Provides an opportunity for students to learn core skills, allowing students to focus on deeper aspects of experimental work during practical sessions Lets students know what is expected of them and what their strengths and weaknesses are Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Reflections Time consuming Human resource intensive Valuable tool: –Assess core skills –Highlight deficiencies –Focussed one-to-one teaching –Can be tailored to assess learning outcomes Students think that OSTEs should be used formatively in every module they study Nandini Alinier, EE2006, 25 July, Liverpool, U.K.

Thanks for your attention Any questions? Nandini Alinier, EE2006, 25 July, Liverpool, U.K.