Evaluation without judgement

Slides:



Advertisements
Similar presentations
Educational Supervision & Find Your Way Around in the E-portfolio Dr Jane Mamelok RCGP WPBA Clinical Lead.
Advertisements

School Based Assessment and Reporting Unit Curriculum Directorate
Exploring the changing attitudes of TESOL teachers to classroom-based research Tim Graham & Alice Oxholm, Sheffield Hallam University.
© Myra Young Assessment All rights reserved. Provided for the use of participants in AM circles in North Lanarkshire Council.
Wynne Harlen. What do you mean by assessment? Is there assessment when: 1. A teacher asks pupils questions to find out what ideas they have about a topic.
Victorian Curriculum and Assessment Authority
Summative assessment and the pre- sessional EAP Conference : Assessment in EAP – what’s the score?, University of St. Andrews, 28 February, 2015 Workshop.
Evaluating Teaching and Learning Linda Carey Centre for Educational Development Queen’s University Belfast 1.
Workplace-based Assessment. Overview Types of assessment Assessment for learning Assessment of learning Purpose of WBA Benefits of WBA Miller’s Pyramid.
Clinical Examination and Procedural Skills The assessment of psychomotor skills in WPBA for the MRCGP examination.
Consistency of Assessment
Lesson planning? It can’t be that difficult! Svetla Tashevska, NBU.
Consistency of assessment Technology subjects (7-12)
6 th semester Course Instructor: Kia Karavas.  What is educational evaluation? Why, what and how can we evaluate? How do we evaluate student learning?
Aspects of professional learning; a new framework for student reflection Rachel Lofthouse & Roger Knill, School of Education, Communication & Language.
The Ofsted ITE Inspection Framework 2014 A summary.
Becoming a Teacher Ninth Edition
Dr Elena Luchinskaya, Lancaster University/ Leeds Metropolitan University, UK.
SOLSTICE Conference th & 5 th June 2015 Transactional Distance and Flexible learning Dr John Bostock Edge Hill University.
Classroom Assessments Checklists, Rating Scales, and Rubrics
Workshops to support the implementation of the new languages syllabuses in Years 7-10.
How does the definition match your thoughts? How does the definition differ from your thoughts? Assessment Definition Assessment is the ongoing process.
Assessment and Reporting in the Primary School Wednesday 29 th October 2014.
Wickford Teaching School Alliance. Our partners 3 Secondary Schools: 1 Outstanding(Beauchamps High School), 1 Good(Moulsham High School) & 1 RI (Bromfords)
Evaluation, Testing and Assessment June 9, Curriculum Evaluation Necessary to determine – How the program works – How successfully it works – Whether.
THE TRAINING AND ASSESSMENT TOOLKIT: A GUIDE TO ACCURACY UEL MENTOR TRAINING 3 JULY 2015.
Link Tutor Update 17 th March Inspection of Leeds Beckett ITE We anticipate being inspected this year as an ITE provider The Ofsted window opens.
The Teachers’ Guides are the core of the programme: Learning progressions by outcomes Support for using Student Activity Books Planning support Links to.
Training Week, August 2016 Assessment for
EVALUATING EPP-CREATED ASSESSMENTS
Classroom Assessments Checklists, Rating Scales, and Rubrics
Subject specialist teaching
School – Based Assessment – Framework
How can we facilitate and encourage research in EAP Teaching Centres?
NEEDS ANALYSIS.
Support for English, maths and ESOL Module 12d: Developing functional mathematics with vocational learners - training the trainers.
Assessment of Learning 1
Assessment and Feedback – Module 1
Assessment brief Post graduate route.
NQT Mentor and Tutor Seminar
SPHERE Study Visit: University of Edinburgh (October 2017)
Designing a Tailor-Made Classroom Observation Program
Classroom Assessments Checklists, Rating Scales, and Rubrics
Link Tutor and Mentor Development Meeting
European Network on teacher Education Policies
The TEAP Portfolio Award and the EAP Teacher Competencies
The Role of Classroom Observation
TALIF Research Project – University of Essex BALEAP PIM at Edinburgh
Confidence in Competence
Critic or critical friend? The case for peer observation of teaching
Dr Claire Kotecki (STEM) & Dr Prithvi Shrestha (WELS)
Assessment of student achievement of outcomes from the 7-10 Science syllabus Use this as a screen saver.
An Introduction to e-Assessment
Assessments TAP 1- Strand 5.
Sarah Lucchesi Learning Services Librarian
Professional development
14 The Role of Assessment. 14 The Role of Assessment.
Tasks & Grades for MET2.
Exploring Assessment Options NC Teaching Standard 4
Tasks & Grades for MET3.
Tasks & Grades for MET5.
Unit 7: Instructional Communication and Technology
Tasks & Grades for MET4.
Caroline Fletcher Innovations in developing teacher assessment literacy: A scholarship circle model Caroline Fletcher
Catherine Beswick University of Nottingham April 2019
Designing Your Performance Task Assessment
Jennifer Rodriguez TB610-60
Providing feedback to learners
Why do we assess?.
Presentation transcript:

Evaluation without judgement Evaluation without judgement? Exploring alternative approaches to EAP lesson observation David Jay Language Services Unit, Anglia Ruskin University BALEAP PIM, University of Edinburgh, March 18th 2017

Objectives Reflect on the use of lesson observation as an evaluation tool in EAP Share experience of piloting a criterion-based framework in peer observation Exchange questions about the proper role of evaluation in EAP lesson observation

Evaluating EAP We have many ways of measuring the effectiveness of EAP provision: Student output (e.g. Crosthwaite, 2016) before and after an EAP writing course, or looking at cohort progress on degree programmes Feedback from students and/or staff (e.g. Salter- Dvorak, 2016) Lesson observation (Hedge, 2000)

? 1. What role (if any) should direct observation of classes play in the evaluation of the effectiveness of EAP courses? 2. If observation can be used to evaluate course effectiveness, what form should it take?

Methodological difficulties ‘Internal paradox’: not all aspects of teaching are observable (Bar-Hama and Leshem, 2008, p.264) Observer’s paradox Doubt about the ‘validity and reliability of observation as a method of assessment in “snapshots” or isolated, episodic performances’ (O’Leary, 2013, p. 7) Practical difficulties of setting up observations compared to processing assignments / feedback

A controversial issue O’Leary: ‘widely debated and hotly contested’ about which practitioners feel ‘very strongly’ (2013, p.6) Dissatisfaction with ‘graded models of assessment’ Damaging effect of ‘outsider’s judgement’ (Malderez and Bodóczky, 1999, p.121)

Developmental Evaluative Describes action/behaviour Judges action/behaviour Serves a ‘mirror’ for the observer or observee Used to assess quality of teacher performance Can be performed by peers or trainees Usually performed by a ‘superior’ Requested by observee Occurs ‘whenever a supervisor wants to’ (Tanner and Green, 1998)

Observation Procedures Evaluative university-wide teaching review: effective, but not EAP-specific; generic criteria not linked to specialised areas of EAP expertise Peer observation based within the Language Services Unit (LSU), explicity non-evaluative; entirely separate from teaching review Existing peer observation format is fully flexible: areas of focus are defined by participants, not criterion-based, with open space for free-text reflections

Selection of criteria TEAP framework ideal basis for a focus on EAP-specific competences (BALEAP, 2008; Alexander, 2013) Fits well with our in-sessional and final phase of pre-sessional; some competencies less easily matched to early stages of our pre-sessional courses Examples: planning and drafting larger summative tasks; investigation of disciplinary differences; analysis of academic genres

Supplementary criteria ‘Critical success factors in EAP teaching’ (Alexander, 2012, p.106-108) Derived from sets of teacher beliefs with a specific focus on ‘low level learners’ of English ‘Students should be challenged with new and unfamiliar material and difficult tasks’ Challenge students with new and unfamiliar material and difficult tasks ‘Academic texts from textbooks can be adapted for this level’ Adapt academic texts from textbooks for this level

Evaluation methods Free-text comments for qualitative evaluation ‘Met/not met’ grading system from teacher training (e.g. CELTA); less appropriate for ongoing CPD mentoring with experienced practitioners Numerical scales: problematic, as would need separate descriptors

Consultation Observation Tool (COT) Used to assess GPs in training (adapted from RCGP, 2014) against a fixed set of performance criteria

Peer observation pilot Pre-observation Observation Post-observation Observee selects 3 criteria to work on Self-evaluation by observee on these criteria Observer makes notes on the selected criteria only. Sharing of reflections Comparison of assessments by observee and observer Discussion of criteria

Use of new criteria very useful: allow specific focus on EAP practice, but give observee control of selection Less realistic to apply a reliable evaluation ‘grade’ for each criterion, because of variable interpretation of descriptors (Thaine, 2004), e.g. precise meaning of ‘autonomy’ Free-text comments gave an effective balance between totally flexible developmental observation and judgemental ‘grading’ Helpful combination of self- and peer-assessment (Cosh, 1999) Initial reflections

Conclusions Qualitative free-text comments, referenced to pre-selected criteria, can be very helpful and are compatible with CPD Evaluative ‘grading’ of observations may lead to difficulties (c.f. O’Leary, 2013) If observation is to be used for EAP course evaluation, other measures should also be taken into account, e.g. student output, feedback

Questions for you! What role does observation play in your context? Is it ‘evaluative’ or ‘developmental’? How comfortable would you feel if your EAP teaching were to be evaluated? Would you want to be able to select the criteria to be focused on? Do you think observation should be included as one of a range of measures of overall EAP course effectiveness? Can peer- and self- evaluation be useful alternatives to top-down evaluations by line managers/course leaders?

References Alexander, O., 2012. Exploring teacher beliefs in teaching EAP at low proficiency levels. Journal of English for Academic Purposes, 11, pp.99-111. Alexander, O., 2013. The leap into TEAP: the role of the BALEAP competency framework in the professional development of new EAP teachers. Professional and Academic English: Journal of the IATEFL English for Specific Purposes Special Interest Group. vol. 42/2, Garnet Education, Reading, UK, IATEFL English for Specific Purposes SIG Conference 2010, Ankara, Turkey, 18-19 June. BALEAP, 2008. Competency Framework for Teachers of English for Academic Purposes. [pdf] Available at: https://www.baleap.org/wp-content/uploads/2016/04/teap-competency- framework.pdf. Bar-Hama, R. and Leshem, S., 2008. Evaluating Teaching Practice. English Language Teaching Journal, 62/3, pp.257-265. Cosh, J., 1999. Peer Observation: A Reflective Model. English Language Teaching Journal, 53/1, pp.22-27. Crosthwaite, P., 2016. A longitudinal multidimensional analysis of EAP writing: Determining EAP course effectiveness. Journal of English for Academic Purposes, 22, pp.99-111.

References (contd.) Hedge, T., 2000. Teaching and Learning in the Language Classroom. Oxford: OUP. Malderez, A. and Bodóczky, C., 1999. Mentor Courses: A Resource Book for Trainer- Trainers. Cambridge: CUP. O’Leary, M,, 2013. Developing a National Framework for the Effective Use of Lesson Observation in Further Education. UCU Project Report. [pdf] Available at: http://www.ucu.org.uk/media/6714/Developing-a-national-framework-for-the-effective- use-of-lesson-observation-in-FE-Dr-Matt-OLeary-Nov- 13/pdf/ucu_lessonobsproject_nov13.pdf Royal College of General Practitioners, 2014. Sample COT Form. [pdf] Available at: http://www.rcgp.org.uk/gp-training-and-exams/mrcgp-workplace-based-assessment- wpba/cot-for-mrcgp-workplace-based-assessment.aspx Salter-Dvorak, H., 2016. Learning to argue in EAP: Evaluating a curriculum innovation from the inside. Journal of English for Academic Purposes, 22, pp.19-31. Tanner, R. and Green, C., 1998. Tasks for Teacher Education. Harlow: Longman. Thaine, C., 2004. The Assessment of Second Language Teaching. English Language Teaching Journal, 58/4, pp.257-265.