Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena Harper, Linda Li, Sue Prentice

Background of our study Individual Consultations (ICs) are still core ASC business Increasing demand for (ICs) at UC but decreasing capacity to meet this demand Need to ensure quality & effectiveness of ICs Inspiration from Stevenson & Kokkinn’s (2009) AALL paper on evaluating ICs

Purpose Develop, trial and analyse the effectiveness of a set of instruments for evaluating individual consultations Secondary purpose – to enhance our teaching and student learning in ICs

The research design Peer observations Student evaluations Reflective practice

Methodology Instruments: Peer-observation (PO) tool Self-reflection (SR) tool Student questionnaire (Q-tool) Participants: 7 ASC staff; Students Time: 4 week period - Semester 2, 2011

The research process Pre- PO PO IC Post PO SR Student Eval Anonymous student feedback PO checklist de- identified & stripped of comments Responses to request for contributions about SR process (non-identifiable) Shared data

The peer observations: a collaborative model A structured approach Establishing a set of principles and criteria inclusive, staff active involvement, non- judgmental, a collaborative process a collaboratively developed peer-observation tool, scheduled pre- and post- observation discussions integrated with self-reflection

Peer-observations: positive outcomes Enthusiastic staff participation Sharing of approaches to good practice and strategies Strengthening the collegiality of the team Benefits of learning from each other

An example of staff comment Thank you for the opportunity to observe your consultation. In spite of my observer status, I was actively involved in the learning and teaching process as I observed its development as the consultation unfolded. The consultation made an important contribution to my learning and professional development by stimulating reflection on my own teaching.

Peer-observations: challenges and constraints Some initial staff reservation The limiting effect of the peer-observation tool The constraint of time

The self-reflection process: reflective dialogues with self and others Scheduled self-reflection time A guided approach with a self-reflection tool aligned with the peer observation tool Linked to the peer observations through post-observation discussions with colleagues Team debrief session to reflect on the whole process

Self-reflections: positive outcomes The self-reflection tool served as a reminder or structural tool Self-reflection was stimulated by observing another and through peer-exchanges. Reflection as a process with ongoing informal discussions

An example of staff self-reflection One thing I would remind myself in future sessions is not to jump to conclusion too quickly… It was only after listening to his explanations that I realised I made an assumption too quickly without getting the real picture! This reminds me of the importance of listening to the student, not just at the beginning of the session, but throughout the session, and the importance of holding back my own judgment and letting the student make his/her own decision.

Self-reflections: challenges and constraints Some staff criticism of the self-reflection checklist Need to modify the tool ‘Self-reflection can’t take place in a vacuum’

An example of staff self-reflection By and large, my reflection took place while I was observing other staff. Upon seeing a particular approach, my instinct was to consider how I would approach the situation if I was the adviser. I learned a lot of useful strategies this way…I’m not sure I can apply ‘reflection’ to my own teaching in a sort of vacuum. When I try to do this, I am only evaluating how I fell short of my own tried and tested approaches. I feel it’s difficult to improve this way. I feel I need to see alternative approaches in practice so I have points for comparison.

The student questionnaire: focus on student learning An improved version: more than quality assurance Encourage the students to reflect more explicitly on their learning

The student questionnaire: focus on student learning Anticipated learning IC Actual learning Evaluation of learning Evaluation Of teaching

The student questionnaire: positive outcomes A constructivist view of learning Student expectation and student learning Focus on most needed aspects of learning

Student comments I am very happy with consultation. It gave me a lot of confidence. Thank you for giving me the chance to correct my assignment. I am happy with the ASC service and it helped me to improve my knowledge and my result. Thank you for all your help it has greatly advanced my grades and understanding!!

The student questionnaire: challenges and constraints Focus on what rather than how and how well Unlinked to peer-observations and self- reflections

Have we achieved our goals? Peer observations Student evaluations Reflective practice

Have we achieved our goals? We successfully developed, trialled and analysed the effectiveness of Q-tool, PO & SR The Q-tool contributed to greater student awareness of their own learning, but did not add significantly to feedback on our teaching PO and SR contributed to better shared understanding of ICs and improved IC teaching practices

What have we learnt from this project? Better, shared teaching strategies for ICs We learnt more from observation than being observed or reflection Necessity to link PO and SR to improve IC teaching practices Using evaluative instruments such as PO and SR requires training and experience

What next? Plan for next round of PO & SR: Refine the PO, SR & Q-tool instruments Pilot a focus group of students to elicit more in- depth comments/suggestions on ASC’s ICs Keep an on-going record of effective IC teaching strategies

Acknowledgements We would like to thank our colleagues Emmaline Lear, Sharon Gibson, Jo Caffery, Ros Byrne and all the participating students for their contribution to this project.